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This is the Technical Appendix to Judgment (No.6) which deals with 
the Horizon Issues. The purpose in preparing this is to avoid what 
is already a long judgment being vastly increased in length by 
including technical IT details within it, which are not necessary for 
the vast majority of readers to understand. 


This appendix has the following structure: 


The History of the Horizon System 
Structure of Horizon generally 
Legacy Horizon and Horizon Online 
Submissions and Evidence 

Bugs, errors and defects 
Conclusions on Technical Issues 


The contents of this appendix are predominantly drawn from the 
two expert reports, as well as the factual evidence, but also the 
parties’ extensive submissions and some of the contemporaneous 
documents deployed in the Horizon Issues trial. Given the size of 
the system, and its evolution into Horizon Online from the days of 
Legacy Horizon, it would have been entirely possible to draft an 
appendix that dwarfed the length of Judgment (No.6). In order to 
avoid that, as such a lengthy document would not be proportionate 
or reasonable, I have summarised certain aspects of the technical 
evidence and documentation where necessary. There may therefore 
be some passages in this appendix which, for example, to someone 
vastly experienced in writing Oracle code, may not read as being 
100% technically accurate or which seem to be a superficial 
summary. However, in my judgment, such lack of full technical 
accuracy in description or terminology does not affect either the 
consideration of the evidence, or the conclusions, or my findings on 
the Horizon Issues. 


The History of the Horizon System 

There was a great amount of agreement between the experts on 
both the history of Horizon and its architecture, both Legacy 
Horizon and Horizon Online. 


Originally, in the 1990s, there was an intended tri-partite project 
between the Post Office, the government department responsible 
for welfare benefits, and ICL. During evidence the second of those 
two parties was often referred to as the Department of Work and 
Pensions, or the Benefits Agency. In fact, prior to 2001 the relevant 
department was the Department of Social Security or DSS (which 
had been created in 1988 from its predecessor, the Department of 
Health and Social Security or DHSS). The Department of Work and 
Pensions was not created until 2001. The different terminology 


does not matter, other than for the sake of accuracy. At least part 
of the reason for the project was so that benefits claimants could be 
paid by using a card, rather than what used to be the case with 
paper books which a branch Post Office would stamp, remove and 
exchange for cash. 


The tri-partite project was called Pathway. ICL Pathway (the 
department within ICL formed for the purpose) was awarded a 
contract for an electronic way of paying benefits, the Benefits 
Payment Card, in May 1996, and there was a Horizon Pilot that was 
introduced in a small number of branches in 1996. This tri-partite 
scheme was abandoned in 1999. Pathway cited “greater than 
expected complexity” and “...major implications for the degree of 
difficulty of the project”, quotations included in Mr Coyne’s report, 
as ultimately leading to the failure of the project. In July 1999 Post 
Office Counters Ltd (or POCL, the previous name of what is now 
the Post Office Ltd) and Pathway agreed to utilise the project to 
automate branch Post Offices. This is what is now called the 
Horizon System or Legacy Horizon, and it was introduced (the 
technical term is “rolled out”) from late 1999 onwards. 


Anyone with a modern smart phone will know that modern IT 
technology is marching ahead at an increasingly ferocious pace, 
and particularly those adept at its use now (including the younger 
generations) may be surprised that so much of what is now dealt 
with electronically, used to be done using paper. Prior to the 
introduction of Horizon, the branch Post Office system used a 
system of paper-based accounts. A paper-based system was used by 
SPMs to account to the Post Office for the branch activity, although 
some SPMs used software packages to assist them in this. Although 
Horizon is used as an accounting system - it is one of its main uses 
- there is more to it than that. However, computerised accounting 
systems have been in use since the 1950s, and Dr Worden 
explained that accounting was one of the “most mature 
applications” for computers in business. This means that as a 
computing application, accounting has been such an application for 
a long time, even in the late 1990s. 


Dr Worden explained the origin of double-entry bookkeeping and 
the principles that underlie it, and explained the process of manual 
bookkeeping, even (in footnote form) as far back as the clay tablets 
of Ur, a Sumerian city-state in ancient Mesopotamia which dates 
from 3800BC. The cuneiform tablets of Ur, as he pointed out, were 
not double-entry bookkeeping, which was invented in the 14" 
century. They were single-entry; an entry would be recorded once. 
Double-entry bookkeeping means that discrepancies can be 
discovered much more speedily, and easily, than otherwise. I 
consider Dr Worden’s explanation of this to be sufficiently concise 


and helpful that I will reproduce parts of it from one of his 
appendices: 


“12. Because any accounting system is intended to track external 
reality, and to give the most accurate possible picture of that 
reality, it is essential from time to time to check the picture of 
reality, held in the accounting system, against reality itself - the 
physical assets of the business, its money in cash or banks, its 
obligations and debts. 


13. However, checking against external reality is (or was) an 
expensive process. You cannot simply look at the books - you have 
to get up from your desk, go out into the warehouse and count 
stock, check your bank balance and count your cash, consult other 
people, and so on. Because checking against reality was an 
expensive process, it did not get done very frequently. If the check 
is only made occasionally, and mistakes are found, the interval of 
time in which one or more mistakes might have occurred is a long 
one. The error is more likely to have arisen from multiple mistakes. 
Looking for several mistakes together is much harder than looking 
for one mistake; there is no ‘telltale number’ to look for. The 
process of ‘drilling down' to find the origin of a mistake is difficult 
and unguided, with few clues. 


14. Double entry bookkeeping changed this. If a bookkeeping 
mistake is made, that mistake will lead to a discrepancy against 
external reality, which will eventually be found in the external 
reality check. But any mistake is most likely to have occurred in 
one column of figures, without any balancing mistakes on the other 
columns. So, the mistake will immediately destroy the trial balance. 


Checking the trial balance is much easier and cheaper than 
checking against external reality. It can all be done by sitting down 


at a desk with the books and an abacus or calculator - so it can be 
done much more frequently. When a discrepancy is found, it is now 
much easier to drill down and find its cause. For instance, if each 
entry in the books is dated (as it will be), by just inspecting the 
books you can find the exact date and nature of the transaction 
which was not recorded as zero-sum, and which destroyed the 
balance. 


15. So double entry bookkeeping immediately reduces the cost of 
keeping accurate and trustworthy accounts. It was an early, and 
highly effective, form of error repellency - in a time when manual 
errors of bookkeeping were likely to be frequent. The error 
repellency guarded against a single point of failure (i.e. a mistake 
in a single column of figures) by making any such mistake rapidly 
and obviously visible, in the next trial balance.” 


(emphasis added) 
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A very simple example was given by Dr Worden in his Appendix B, 
which again I will reproduce. For those with no experience of the 
pre-decimal system the units of money might be a little vague, but 
the principle applies whether one expresses the money in pounds 
or shillings: 


“8. Double entry bookkeeping can be understood by starting with 
the simple case of a trader on a farmer's market, who has some 
money, and who has some sheep. He can keep a list of his money, 
and a separate list of his sheep - complete with their names if he 
wants to. He can track changes to those lists, with entries like 
'23rd July: sold one sheep - Dolly’. From the changes he can work 
out his current position, in money or in sheep: 'now I have 17 sheep 
left'. This is single-entry accounting. 


9. Double-entry bookkeeping starts when his list of sheep contains 
two types of information - the sheep he has, and the price he paid 
for them; and he also keeps a separate list of his money. He tracks 
the changes to these lists in a series of dated transactions: '25th 
July: bought one sheep for 5 shillings'. With that transaction, the 
sum of his money list goes down by 5 shillings, and the money total 
of his sheep list goes up by 5 shillings. So, he makes two entries - in 
those two lists - with money value -5 shillings and +5 shillings, and 
the total money value of the two lists does not change. Because of 
the self-consistency of mathematics, however he chooses to do 
those two sums, the sum of the two sums should not change from 
day to day. This is his double entry ‘trial balance’. If the number 
does change from any day to the next, he knows he has made a 
mistake - and he can start to track it down.” 


This simple example can be applied to any stock held in any 
business, in this litigation branch Post Offices, and the sum of 
money which is used to purchase stock in a branch by a customer. 
To return to the well-worn example of the book of 1* class stamps 
used earlier in the litigation, if a SPM sells one book of these to a 
customer for cash, the cash position of the branch improves by the 
cost of that book of stamps, paid to the branch by the customer (as 
of today, £8.40 for a book of 12 1*' class stamps) and the stock of 
books of stamps held in the branch reduces by one. This is a simple 
example because the stamps are part of the branch’s stock. There 
are more complex transactions in a branch that do not involve 
stock actually held within the branch; this will usually be because 
the branch offers services and products on behalf of the Post 
Office’s numerous clients, that are not stock in the traditional sense 
physically held within that branch. 


Dr Worden also explained the advantages of double-entry 
accounting, which include reduction of fraud and theft. 
Notwithstanding the differences to which accounting systems are 
put (or to put it another way, how the information held within such 
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systems is used), such as management accounting (for those 
responsible for making management decisions about the business 
in question) or financial accounting (for example, for departments 
who have to declare profits and account for taxation) both sets of 
information are subject to audit, which is a word which means 
review, examination, inspection and appraisal. In lay terms, it is the 
checking of the information held within the accounts. 


The word “audit” is used in different ways, and for different things, 
in this group litigation. The Post Office employs a number of people 
called auditors to check on the accounts of branch Post Offices. 
When Mr Bates, for example, started as a SPM, the Closing Day 
Audit was conducted on the branch (run by his predecessor) as part 
of what occurred on what was called Branch Handover Day. This is 
done so that a SPM on their first day starts with particular stock 
and cash, identified by the Closing Day Audit, and their branch 
activity uses that as a starting point. Some SPMs may be 
suspended, or have their engagements suspended, as a result of 
what the Post Office (through its auditors) uncovers at other audits, 
which may be performed at any time (which is entirely 
understandable, and is included in the terms of the SPMs’ 
contracts). These audits are done to check the state of the branch 
accounts, and the cash and stock held in the branch. 


The Post Office itself is also audited by its auditors (such as Ernst & 
Young or E&Y), and the Horizon System is itself audited in the 
sense of checks being performed that it complies with various 
standards. Audit is also used as part of the expression “audit data”, 
further explained in Part K of Judgment (No.6) itself. 


The financial accounting functionality of the Post Office was 
provided by something called SAP, which is an industry leading 
resource planning and accounting system. From 2004 onwards this 
was by something called POLFS, and in 2010, with the introduction 
of Horizon Online, this became POLSAP. POLSAP was created by 
merging two other systems, POLFS and SAP ADS, the ADS in that 
latter acronym standing for Administrative Data Services. Horizon 
provided mainly two things; management accounting functionality, 
as well as Electronic Point of Sale functions for the branches (also 
called EPOS). There is, in Horizon Online, overlap between Horizon 
and POLSAP. Dr Worden stated: 


“I understand that the financial accounting functionality for the 
Post Office was provided by SAP over most of the disputed period 
(from 2004 by POLFS, a SAP application, which in 2010 merged 
with SAP ADS to form POLSAP), whereas Horizon provided mainly 
management accounting functionality, as well as Point of Sale 
functions for the branches. This dual nature required close 


integration between Horizon and the SAP systems, so that pictures 
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of financial reality given by the two systems were at all times 
mutually consistent. There is a large overlap between the 


information stored in the two systems, which will be described 
below.” 


(emphasis added) 


Some computers provide a highly complex computation function. 
Dr Worden stated, and I agree with him, that the functions of an 
accounting system include the input, secure storage and output of 
many types of information, in large volumes (many transactions per 
day), with rather little computation. By far the most important type 
of computation that occurs in financial accounting is 
straightforward summation of numerical quantities such as money 
and stock, subject to the rules of double entry bookkeeping. For 
Management accounting, some other kinds of computation are 
needed, such as for forecasting purposes. However, these are not 
usually computationally demanding or complex. This is not 
disparaging of those who designed the system, but given the 
function of Horizon so far as branch Post Offices are concerned, 
Horizon does not possess (nor does it need to possess) a highly 
complex computational function in the sense that very difficult and 
demanding complex mathematical equations have to be processed 
at high speed, for example. The computers at NASA in the United 
States that are responsible for space exploration, and control of 
satellites, will be dealing with far more complex computation in 
terms of mathematical function; however, rarely would there be 
12,000 moon shots underway at once. Horizon deals with a huge 
number of transactions per day (about 6 million) across many 
thousands of branches, but the nature of the calculations is 
essentially arithmetical. This is not to understate the complexity of 
Horizon both in terms of the scale of the system, the number of 
products or the number of different clients with whom the Post 
Office (and hence SPMs in branches, on behalf of the Post Office) 
deal. Fujitsu, in a document of 14 January 2004, described Horizon 
as ‘Europe's largest non-military IT contract’, so Horizon is at the 
high end of complexity amongst IT systems. Dr Worden stated that 
it represents “many thousands of man-years effort in development 
and testing, and its documentation alone more than 100,000 
documents”. It can therefore clearly be seen to be undoubtedly 
highly complex. 


However, Dr Worden also said this: 


“On the other hand, compared with the IT estates of various large 
organisations I have worked for (such as the NHS, Barclays Bank, 
UBS or RBS), the Horizon system is probably no more complex, and 
in some ways less complex. The banks' IT estates, like Horizon, 
have a complex corporate back-end and an extensive branch office 
network. They were developed over a longer time period (30-40 


17. 


18. 


19. 


years) using development team sizes similar to or larger than 
Horizon, often merging together or integrating the IT systems of 
previously independent organisations. This gave them a degree of 
legacy complexity, and design compromise, and corporate amnesia, 
not found in Horizon. There are parts of these IT estates which 
'nobody dares touch'.” 


This means that simply because Horizon evolved as it has is not an 
answer to the more controversial of the Horizon Issues. Its origins 
as a system to accomplish something else, the payment of benefits, 
its incremental addition of products and clients, and its age, are not 
of themselves indicative of any particular level of performance. 


Structure of Horizon Generally 


Although computerised accounting systems had been used widely 
for some decades since about the 1960s, in the 1980s there was 
what was called a “major step change in their architecture”, when 
relational databases were invented/developed and used. A 
relational database is a digital database based on the relational 
model of data, and was first proposed in 1970 by EF Codd. Mr 
Codd was an English computer scientist and invented the relational 
model for database management. This model uses a structure and 
language consistent with what is called first-order logic, although it 
is also called other things (such as quantificational logic) and this is 
used both in mathematics, computer science and also linguistics. 
Inter alia it allows the use of sentences that contain variables. In 
the relationship model, all data is represented in terms of tuples, 
grouped into relations. In relational database theory, a relation is a 
set of tuples (dı, dz, ..., dn), where each element dj is a member of 
D,, which is called a data domain. Each element is termed an 
attribute value, and an attribute is a name paired with a domain 
(which is also called a data type, or simply a type). In mathematics 
a tuple is a finite ordered list or sequence of elements. 


A database organised in terms of the relational model is called a 
relational database. A software system used to manage relational 
databases is called a relational database management system or 
RDBMS. Many such systems use Structured Query Language or 
SQL to query and maintain the database. A famous and widely used 
RDBMS is Oracle RDBMS, which is sometimes called Oracle 
Database or even simply Oracle. This is a proprietary multi-model 
database management system produced and marketed by the 
Oracle Corporation, a US technology corporation based in 
California. It was first released in 1979. 
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The system software therefore used for a relational database is 
built and supported not by the accounting system developer, but by 
the supplier of system software, which will be a major company 
such as Microsoft or Oracle. The accounting software makes calls 
to the RDBMS to store and retrieve the information. These work (in 
outline terms) by storing structured information. Unstructured 
information can be stored in computer systems generally, but this 
requires very advanced computer programs to understand it. More 
simple programs, such as accounting systems, store and use 
structured information. Structured information is (for example) 
something contained in a spreadsheet, made up of information in 
different cells in rows and columns. Adding up the value of a list of 
transactions can be accomplished simply by adding up the figures 
in a particular column that relates to this information for a number 
of transactions. 


The main use of a relational database is securely to store large 
volumes of structured information. The way it does so can be 
understood as having large numbers (tens, hundreds or even more) 
of different spreadsheets (which are called tables) and which are 
linked to one another. Two different tables in a database are linked 
to one another (in a 'relation') when they both have one or more 
columns with the same meaning and share values in those columns. 


Dr Worden explained: 


“44. A typical relational database may have tens or hundreds of 
separate tables; each table may have tens or occasionally hundreds 
of columns; and a table can have any number of rows, up to 
millions if necessary. 


45. One relational database can act as a 'server' to one or more 
application programs, which are performing actions directly visible 
to their users. The application programs make calls (requests) to 
the relational database management system, which alters or 
retrieves the data to fulfil the requests. The core service provided 
by the relational database is to securely store and retrieve this 
information, for the application programs, or for others who 
retrieve information from the database more directly. Both the 
phrases 'securely store' and 'retrieve' have a lot packed into them. 


46. The phrase 'securely store' implies several guarantees: that 
once an item of information has been stored in the database, it will 
not be lost or unintentionally changed; and that it is a part of a 
consistent collection of information, whose integrity will not be 
compromised in any way.” 


The integrity of information is central to relational databases. 
Indeed, I consider that the integrity of information could be said, in 
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a way, to be central to the concept of the operation of Horizon, if 
not accounting altogether. 


Dr Worden explained the concept of integrity, and integrity 
constraints, in the following way: 


“47. The idea of integrity of information is central to relational 
databases. When the structure (tables and columns) of a relational 
database is first defined (in a relational schema) that schema 
defines various types of integrity constraint on the data, for 
instance that: 


47.1. certain columns in tables are mandatory, and will always be 
given values. 


47.2. There can never be a record in some table (call it table A) 
unless there is a corresponding record in some other table B. For 
instance, there can never be a record of an invoice to a customer, 
unless a record exists for the same customer. The invoice table and 
the customer table both have a column ‘customer id'; for every 
invoice record, there must be a customer record with the same 
customer id. 


48. The database management system guarantees that these 
integrity constraints will be true for all time. If any application tries 
to make a change to the database which would violate an integrity 
constraint, that change is rejected by the DBMS, and no change is 
made at all. One change to the database may involve changes to 
several tables at once, so that after all the changes are made, the 
integrity constraints are still true; but part way through the 
changes, the constraints are temporarily untrue. One such package 
of changes, involving changes to one or more tables, is called a 
database 'transaction'. The DBMS guarantees that: 


48.1. After any completed transaction, the database will still obey 
all its integrity constraints. 


48.2. After any completed transaction, the changes made to the 
database can never be lost - even in the event of hardware failures; 
there are robust ways to recover the information. 


48.3. If a transaction would violate an integrity constraint, it is not 
allowed by the DBMS; no changes will be made to any table, so the 
database will still be in a consistent state, as if the change had 
never been requested. 


48.4. When one application is making multiple changes to the 
database in a transaction, so that the database is temporarily in an 
inconsistent state (when some but not all of the changes have been 
made), those changes are never visible to other applications or 
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users before they have all been completed. Other applications and 
users can only ever see a consistent state of the database (either 
before all the changes, or after all changes), in which all the 
integrity constraints are true. 


49. This set of guarantees, given by the DBMS, is called 
‘transactional integrity’. It has been built into the fabric of all 
relational databases and has been relied upon by thousands of 
applications which use relational databases, since the 1980s. 
Builders of applications can have a very high degree of confidence 
that their DBMS will maintain transactional integrity. Builders of 
accounting systems have relied on that guarantee.” 


Relational databases support the SQL language, and small numbers 
of records can be retrieved from within application programs from 
a limited number of linked tables, and the records are filtered 
based on data values. Dr Worden considered that the possibility of 
there being a bug in Oracle to be “extremely remote”, and he 
ignored it. As he put it: 


“51. SQL can be used directly by end users, without the use of any 
application program, to selectively retrieve and display records. 
However, it is more common for the users to use a general report 
writing tool, which not only retrieves the required records, but 
formats the information in easy-to-read reports - with appropriate 
column headers, formatting of fields, grouping of records, 
computations of sums of groups of records, and so on. Nearly all 
the reports needed from an accounting system are created using 
these report writing products, which can be rapidly configured to 
produce any new kind of report, either one-off or regularly as 
required. 


52. All these capabilities of a relational database have been present 
in relational database products such as Oracle since the 1980s and 
have been tested by possibly millions of different applications 
which use those capabilities and rely on them. So, when I am 
discussing the issue of bugs in an application such as Horizon, the 
possibility that these bugs arise from bugs in the underlying DBMS 
- particularly when it is the world market leader Oracle - is 
extremely remote, and I shall ignore it.” 


This is a similar approach to that adopted by Mr Godeseth when he 
was giving his evidence about PEAK 0195561, when a problem was 
reported to the SSC on 4 March 2010. This was where a SPM had 
tried, on 2 March 2010, to transfer out £4,000 (referred to in the 
PEAK as 4,000 pds, which means either pounds (plural) or pounds 
sterling) from an individual stock unit into the shared main stock 
unit when the system crashed. Mr Godeseth had been looking for a 
bug at the time of a “red alert” during the early days of Horizon 
Online, and his evidence about this is included at [341] to [349] of 
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Judgment (No.6). He referred to one possibility being a bug within 
Oracle, which is dealt with at [350] of that judgment, which he said 
“would be huge” and which he also said “was not the bug that I 
was looking at for the red alert”. 


For the avoidance of doubt, when the experts reach agreements, or 
where I make any findings, about the presence of bugs, errors and 
defects in the Horizon System (either Legacy or Online) either in 
this appendix, or in Judgment No.6, these are not made concerning 
exactly where within the different programs or applications within 
the Horizon System such bugs, errors or defects are present, or by 
apportioning responsibility to any particular software provider. The 
Horizon System includes Oracle as the RDBMS, but there are a 
considerable number of other companies involved. Those 
companies are not represented and that is not what the Horizon 
Issues trial is about. The experts would sometimes refer to what 
were called “sub-contracting parties” but it was not clear to me 
whether they were sub-contracted either to Fujitsu or to the Post 
Office. Nor is it relevant to the Horizon Issues whether the fault is 
within a particular layer within the system. The relevant issue is 
whether there are bugs, errors or defects within Horizon. 


The software in Horizon is what is called data-driven. If the 
accounting application software was written in a way that 
depended directly on the accounting needs of a client (here, the 
Post Office, what Dr Worden called a “chart of accounts”), then 
each company would need different accounting software. This 
would be very uneconomical (in writing and testing all that 
software), and this is not the way in which software is developed. 
The chart of accounts is not 'hard coded' into the application 
software. The software is written to work with any valid chart of 
accounts; and the chart of accounts itself is treated as data and 
stored in the database. In this way, changes in the chart of 
accounts (which occur from time to time) do not require changes in 
the accounting software. 


“59. This is one example of data-driven software - in which any 
requirement which might change frequently is encoded as data, 
rather than software code. The code is written and tested to work 
with all allowed values of the data, to meet a wide range of 
requirements by changing the data rather than the code. The 
modern practice of software engineering is to use data-driven 
software as far as possible. This is not always as far as one might 
like; some differences between companies and their requirements 
are best expressed as differences in code, rather than data”. 


Accordingly, the accounting application or applications are built as 
layered software architecture. This means that the accounting 
application is built “on top of” the relational database. There will 
usually be a minimum of three layers in a typical system of this 
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type. Layers are also sometimes called tiers. These are the user 
interface layer, which presents information to users, and accepts 
their inputs. This is the layer that a SPM would both observe, and 
which would react to the actions of the SPM (or their assistants) in 
the branch such as pressing buttons at the terminal for various 
transactions. The second layer is the business logic layer, 
responsible for carrying out business processes, and supporting 
users in doing so. The third layer is the data layer, responsible for 
storing and retrieving data. More complex architecture has more 
than these three layers, and there may even be layers within layers. 
In the case of Horizon, the user interface also includes a Point of 
Sale interface which is used in the branches. This interface can be 
thought of as part of the Horizon Point of Sale application, rather 
than the Horizon accounting application. 


Almost all complex IT applications are designed in this way, in 
order to isolate different kinds of complexity within different layers, 
and to reduce the possibility of unwanted interactions between 
functions in different layers. A typical ‘client server’ layering 
structure would include at the very least the three layers identified 
above (a user interface layer, a business logic layer, and a data 
layer) but the layering in Legacy Horizon is more complex than 
this. 


The main purpose of defining an architecture in layers is to 
separate the functionality into parts in the different layers, with 
well-defined and simple interfaces between the layers. This not only 
makes each layer easier to design, build and test; but also, if there 
are errors that are not found in testing, it should be easier to 
understand and isolate the cause of the errors by inspecting the 
exchanges between layers. Dr Worden considered layered 
architecture to be an important countermeasure. It could also be 
seen as an important part of designing such a system. It is part of 
modern practice. 


Legacy Horizon 
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The architecture of Legacy Horizon up to 2002 is described in a 
lengthy document called Technical Environment Description dated 
22 October 2002. This document states: 'The system architecture 
adopted to meet these requirements is not based on conventional 
client-server models. Nor does it conform to traditional central- 
system models. It adopts an entirely original and highly innovative 
four-tier model that effectively merges the qualities of central 
systems and client server systems'. The diagram that accompanies 
this in the document actually has five layers rather than four. If two 
boxes in the diagram which are entitled 'Agents' and 
'Correspondence' are counted as one 'Agents' layer, which is what 
Dr Worden did, there are four layers: 
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1. Counter. 
2. Agent. 
3. Host. 


4. External Interface. 


POCL 
Clients 


TIP OBCSp SAPADS NBE 


Hosts 


Distribusion 


Courter 


The counter layer consists of all hardware and software in the 
branch. It includes all hardware and software required to support 
the counter activities required for all products and customer 
services offered in the branch. This was largely built based on a 
commercial product, Riposte from Escher Group, a separate 
company from Fujitsu. 


Riposte provided much of the Graphical User Interface (which is 
the basis of all user input and output at the counter) and provided a 
mechanism for secure distribution of messages between the 
branches and the two back-office data centres, which are also 
referred to in some places (including Dr Worden’s report) as 
campuses. These were located at Bootle and Wigan. The message 
distribution passed through the Wide Area Network in the diagram. 


The Correspondence Servers handled communication over the 
network. 


The function of the Agent layer was to provide two-way translation 
of data between the formats used in the counter layer and the 
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network (these formats were described by Attribute Grammars) 
and the formats used in the Host layer. The agent layer is also 
responsible for extracting the Audit of all data passing through the 
Correspondence Servers. 


An Attribute Grammar is a description of a message structure 
which is set out in the same pattern as a tree. Dr Worden used the 
phrase “a tree-like message structure in terms of its parts and their 
sub-parts”. In more recent IT systems, tree-like messages are 
usually sent in XML (Extensible Message Language), with their 
structure defined in a notation called XML Schema. This is used in 
parts of Horizon Online. An XML document is a string of what are 
called characters. These characters are divided into markup and 
content. What is called the processor analyses the markup and 
passes structured information to an application. Generally, strings 
that constitute markup begin with the character < and end with a >, 
or they begin with the character & and end with a ; (ie a semi- 
colon). Strings of characters that are not markup are content. 


Because Legacy Horizon was developed before the use of XML 
became widespread, Attribute Grammars fulfilled this function in 
Legacy Horizon. This is probably the way that Riposte worked. In 
Legacy Horizon, Riposte provided the facility for reliable 
replication of data between the branches and the back-office or 
data centres/campuses. This means that if certain types of data 
were created at the branches, Riposte should have ensured (the 
term used by Dr Worden was “guaranteed”, but I consider that 
overstates it) that the same data would be available at the data 
centres/campuses. Dr Worden accepted that “if the underlying 
network was unreliable, it might take some time for Riposte to 
deliver this guarantee”. He did not say how long that might be and 
he was not asked about it. 


One of the bugs in the bug table was also called the “Riposte Lock” 
problem. This will be further addressed in Part E of this appendix. 
It was a problem with Riposte that was known about at Fujitsu. 


The bulk of the back-office functionality was provided in the Host 
layer. Host applications were typically batch systems, processing 
data in large batches on a daily basis. A complex daily batch 
schedule was used to control the sequence and timing of these 
batch processes, using the Maestro scheduling product (not to be 
confused with the Maestro software, made freely available by 
NASA to allow users to look at photographs and the daily progress 
of the Mars exploration rovers). It was the Host layer (and for most 
purposes, only the Host layer) which communicated with the IT 
systems of Post Office client organisations, through the External 
Interface Gateways. Dr Worden described the three layers of 
architecture which resided at Wigan and Bootle as “the back-end 
architecture”, and these were the Agent layer (which included the 
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Correspondence layer), the Host layer, and the External Interface 
layer. 


a 


The following are shown in the above diagram as elements, and 
some description of them is required. 


RDMS is the Reference Data Management System. Reference Data 
is described further at [52] below. This management system is a 
dedicated IT application which is needed to manage the reference 
data, and to distribute it appropriately to branches. 


The Data Warehouse consists of one or more databases, whose 
structure is designed to support flexible and open-ended querying 
and reporting by the Post Office. Different kinds of information 
which pass through the host systems are siphoned off into the data 
warehouse and stored there. This storage is within data structures 
designed for reporting (which is done by means of querying those 
structures). Functionality which depends on a data warehouse 
includes the Management Information System or MIS and other 
applications such as those which look for unanticipated trends and 
correlations in data. 


MIS is the component built on the data warehouse to provide the 
Post Office with relevant information. The data warehouse and the 
MIS are important parts of the Horizon System. Dr Worden stated 
that “in cases of human error in business processes, operational 
errors in managing Post Office business on Horizon, or software 
errors in Horizon, some resulting discrepancy or aberration will be 
rapidly visible through the MIS. MIS facilities were also used by 
Fujitsu staff. Many pairs of eyes are inspecting the outputs of the 
MIS, in hundreds of different reports or spreadsheets. One purpose 
of this is to ensure the rapid detection and correction of many types 
of errors. These include software errors”. By “pairs of eyes” Dr 
Worden was referring to human intervention or observation. 
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The Transaction Processing System or TPS has as its purpose the 
'harvesting' of all types of transaction taking place in the branches, 
and to pass them on to other IT systems in the Post Office - initially 
to TIP, and later to POLFS when the system was changed. 


Transaction Information Processing or TIP was, as at 2003 (the 
date of the diagram above) the gateway to all other Post Office data 
processing, including accounting. After 2004, Post Office accounts 
were held on an SAP system, which was called POLFS; so TPS 
passed data to POLFS, rather than to TIP. 


The black circles in the diagram denote points, as Dr Worden 
expresses it, 'at which ownership of data conceptually changes and 
hence at which audit information is generated’. 


Dr Worden quotes the 2002 Technical Environment Description 
document, what he refers to as the “the 2003 design document” to 
explain the Host at paragraph 165 of his report. ‘One essential task 
that can only be carried out at the Host layer is reconciliation. The 
Host is the only system component that can detect discrepancies 
between the transactions carried out at the Counter (and hence 
reported back to Post Office Ltd via TPS), and those that were 
authorised or expected. It should be in a position to send 
reconciliation reports back to its Client. These enable the 
discrepancy with the TPS records to be identified and resolved.’. 


He considered that “this reconciliation, carried out in the Host 
layer, is an essential element within [Legacy] Horizon for detecting 
and correcting errors made at the counter (robustness measure 
UEC)”. 


Reconciliation and TCs have the effect of detecting and correcting 
the effects of many possible software errors, as well as human 
errors. Dr Worden was of the view that if there were any such 
software errors, these would probably occur with such high 
frequency, and occur uniformly across all branches, as giving rise 
to so many TCs, that the Post Office would soon suspect a software 
error (for instance, seeing the effect repeatedly in some MIS 
report) and require Fujitsu to correct it. However, this is 
supposition in my judgment. This is because it depends upon a 
software error occurring very frequently, and being noticed 
rapidly. Some accepted errors (for example Callendar Square) lay 
in the system for 5 years. The evidence of fact advanced by the Post 
Office itself, which includes that in respect of the admitted or 
acknowledged bugs, makes it obvious that this supposition is not 
made out. Further, the detail of the numerous PEAKs show that it is 
a combination of factors, including the speed at which a SPM 
performs some tasks, that sometimes led to unexpected events 
occurring. If something happened at a particular point - one point 
put by the Post Office to Mr Tank was that an outage occurred ata 
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very precise moment - then an adverse impact would or may occur. 
That precise combination of circumstances may not occur with the 
frequency that Dr Worden stated. His supposition was that a bug 
would have a wide impact across many branches with a high 
frequency, which would mean that the Post Office would suddenly 
realise its existence. That is not a valid supposition or assumption 
given the evidence in this trial. 


The Horizon System has certain core components. They include 
both Reference Data and Transaction Data. The difference between 
these is as follows. 


Reference Data is effectively data about products and operational 
elements and is a critical element of the Horizon system. Reference 
Data interfaces with a wide range of Horizon components. Without 
Reference Data, Horizon would not fully function, nor could 
Subpostmasters operate their branches. It informs the operation of 
the Point of Sale system at the counter. This is now managed and 
operated by ATOS. Many of the business applications in the 
branches are driven by reference data, and this approach has 
advantages over hard-coding the different business applications, as 
that would require changes to the code if the applications were to 
be changed. It is much more flexible, to manage changes over time 
and across branches to use the reference data approach. This is 
because when changes are made, the reference data is changed, 
but not the code. 


The integrity of Reference Data is critical for the correct operation 
of a variety of systems within the Horizon architecture. The Post 
Office’s Reference Data Management Centre (RDMC) supports the 
loading, storage and release of Reference Data within the Horizon 
system. The Reference Data Distribution Service (RDDS) 
distributes Reference Data to Post Office branches and other data 
centre systems. The Post Office Reference Data Team deals with 
the delivery of Reference Data and verification of operational 
business change through Reference Data. Horizon counter 
Reference Data was distributed by the Riposte messaging facility in 
Legacy Horizon, although this is not used in Horizon Online. 


Mr Coyne considers that there were insufficient appropriate 
change control processes prior to 2017, based on a document he 
has seen which is dated July 2017 which stated “... we have now 
aligned that all Reference Data changes go through the appropriate 
change process”. Mr Coyne takes this to mean that prior to this 
date, all Reference Data was not going through the appropriate 
change process, or that change process did not exist. In my 
judgment that is probably reading too much into that particular 
entry; however and in any event, change control processes require 
two things in order to work effectively. Firstly, they must exist. 
Secondly, they must be applied. 
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Some aspects of how business applications were built on Riposte 
were explained in the expert evidence, without going into 
enormous layers of explanation of technical complexity that were 
unnecessary. These included the following: Zero-sum baskets for 
customers meant that the overall or net impact of all the services 
provided for one customer was a sum of money which the customer 
was required to settle. It was required that the cash or other money 
produced by the customer should exactly match the cost of services 
provided; therefore, the whole basket of services and customer 
settlement had to be what is called “zero-sum”, before the basket 
could be recorded in the branch (and then, through Riposte 
replication, later recorded in the back-office systems). The impact 
of every business application would be fed into the relevant 
systems at the Post Office, typically overnight. These systems were 
operated by double entry bookkeeping. The only way to put 
postings into the accounting system was by double entry, which in 
turn could only be done for zero-sum baskets. Dr Worden 
considered this a robustness countermeasure. 


Any other actions performed in the branches which had an impact 
upon the Post Office’s accounts (which would include stock 
management, cash drawer management, balancing and 
reconciliation) also had to be zero-sum. These could only be carried 
out in the branch in packages of updates which were zero-sum, 
when summed across different Post Office account codes. This was 
another instance of a robustness countermeasure. 


All branch applications had to have what was called transactional 
integrity. This means that all customer business applications, 
balancing and reconciliation, cash management and stock 
management were built so that any zero-sum package of updates 
from those applications would either succeed completely, or would 
fail completely. In the latter case, they were supposed to have no 
impact on branch accounts. This transactional integrity was given 
the acronym TIN, and was supposed to be enforced by the Riposte 
infrastructure. It should have been impossible in the event of a 
failure (such as hardware failure) for a part-completed set of 
updates to be recorded in the branch and then replicated to the 
back-office systems. This was necessary to prevent the accounting 
system from being subjected to non-zero sum updates, which would 
violate its double entry basis and cause later failures of its trial 
balances. 


The only exception to this principle was that of so-called 
'recoverable transactions' - where some irreversible interaction 
with a Post Office client system took place part way through a 
transaction - so it could not be undone in the case of a later failure. 
In these cases, the user on the counter would be guided through a 
short set of recovery steps, to produce a consistent zero-sum result 
which reflected what had happened. As a comment made here by 
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me for convenience, it is the case that some of the evidence of fact 
from SPMs which I have accepted about transactions that gave rise 
to difficulty in branches occurred when there were outages. It will 
be seen from consideration of the bugs in the bug table that these 
were not the only instances when bugs had some impact on branch 
accounts. However, whether there was a hardware failure or 
outage, the system was built so that partly completed transactions 
were not supposed to be capable of entering the accounting 
system. Dr Worden opined that it was, as he put it, “of course, 
possible for the user to make some mistake in these steps, which 
may have been unfamiliar. In these cases, the mistake would often 
be detected later by a reconciliation process, which would typically 
lead to a TC. This robustness measure was a correction of user 
errors (UEC).” However, that statement in my judgment jumps 
straight to a conclusion that user error would be to blame for 
failures in recoverable transactions, and this would “typically lead” 
to the Post Office issuing a TC. In my judgment, the correct 
approach is to consider bugs, errors or defects within the Horizon 
System and whether these would lead to failures in recoverable 
transactions. This is the correct approach to answering the Horizon 
Issues. 


Many applications were not coded individually but were coded as 
generic applications which could be configured to run different 
specific applications by altering reference data, as has already 
been explained. This is called “data-driven software”. These 
applications were referred to in Legacy Horizon as 'soft-centred' 
applications. They have the attraction of adaptability in that the 
data can be changed without new code having to be written. There 
were however still some hard-coded applications in Legacy Horizon 
in any event. Errors could often be corrected more quickly if 
applications are driven by reference data in this way. Reference 
data is much more concise and understandable than code, so it is 
much easier to create it or detect errors in it. Dr Worden 
considered that errors in the underlying generic code would affect 
a set of specific applications, and so should be easy to detect. 
However, this can be seen not to be a statement of universal 
application. The Callendar Square bug was present for 5 years 
before it was properly discovered (although its effects were 
experienced, to use Anne Chambers terms, “most weeks”). 


Reference data could be as simple as lists of available products and 
their prices (which clearly might change frequently), or might be 
more complex - for instance, to describe the sequence of steps 
needed to handle some business transaction. 


The other important component that can be usefully addressed 
following Reference Data is Transaction Data. There are four 
sources of transactions that make up transaction data: 


1. Those manually entered by a user (which will be a SPM or their 
assistant) in branch at the counter; 


2. Transaction Corrections (TCs) which are produced by the Post 
Office to be accepted by a user in branch to correct discrepancies 
in the accounts; 


3. Transaction Acknowledgements (TAs) which are non-counter 
transactions and typically initiate from somewhere else. This is 
from another area outside the Horizon System so far as the Lottery 
is concerned, because the information comes from Camelot relating 
to the lottery sales a branch has performed. These transactions are 
typically relayed to Post Office/Fujitsu and need “accepting” into 
Horizon before forming part of the branch’s transaction data. This 
is done by means of TAs sent to each branch. The SPM does not 
have the option to reject them. TAs did not exist prior to the Ping 
fix; and 


4. Fujitsu inserted transactions. These are injected into branch 
accounts by Fujitsu. They may be performed in order to ‘balance’ a 
discrepancy. These do NOT require acceptance by SPMs in the 
Same manner as TCs and TAs. 


Horizon Counter 
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The user in a Post Office branch, the SPM or assistant, will deal 
with customers and for each transaction press a number of 
different icons in order to compile that customer’s “basket”. As has 
been seen, different products from different clients can be provided 
and sold to the customer in the same basket. These may be paying 
bills, prepayment for services, acquisition of licences (such as a TV 
licence), and money and insurance management. 


The SPM is not concerned that (for example) some of the money 
collected from a particular customer may need to go to DVLA (for 
car tax) or to a utility company (for a gas bill). It is the Post Office 
that accounts to these clients, and it does so centrally and by 
means of a back-office activity. 


As well as these counter activities, Horizon also needs to support 
the periodic process of balancing and rollover for each branch. 
Every branch operates in Trading Periods, which are either four or 
five weeks (according to a timetable published periodically by the 
Post Office). At the start of each Trading Period the branch is 
supposed to be ‘in balance’. This means that the physical stock and 
cash in the branch agrees with the data on stock and cash held in 
Horizon. Then, during each Trading Period, Horizon records all 
customer transactions made at the branch, so it records the 
changes in cash and stock. It also records any replenishments or 
remittances of cash or stock in the branch. Horizon will (or should) 
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record all changes in cash and stock held at the branch during the 
Trading Period, and can compute, from the starting amounts and 
the changes, the expected amounts of cash and stock at the end of 
the period. This leads to something called the Branch Trading 
Statement. 


At the end of each Trading Period, the SPM counts the physical 
cash and stock in the branch and compares it with Horizon's data 
for the same items. This is called 'balancing'. If the numbers are all 
equal, the branch is in balance and is permitted to 'roll over' to the 
next period. If the two sets of numbers are not equal, there is a 
discrepancy between what Horizon considers should be present in 
the branch (for example the actual cash) and what is physically 
present (the actual cash held at the branch). This also applies to 
some stock such as stamps. These discrepancies are at the heart of 
this group litigation, as the reason for the discrepancies are so 
controversial. As part of “rolling over”, a SPM with a shortfall or 
discrepancy has to “make good” or “settle centrally” at the time, as 
part of bringing that Trading Period to an end. Unless they are 
prepared to do that, the next branch Trading Period cannot 
commence. Each of those two acts, making good or settling 
centrally, involved payment by the SPM to the Post Office; the only 
difference is the former means payment immediately, and the latter 
means seeking time to pay. 


Horizon also supports the activities of replenishing stock such as 
stamps, and of replenishing or remitting cash. Accepting a delivery 
of cash from the Post Office into the branch is called, by factual 
witnesses on both sides in the litigation, “remming in” that cash. 
This is probably shorthand for remitting or remittance, but all 
involved in the case call it “remming”. 


Dr Worden used DVLA as an example of how the back-office works. 
He stated: 


“Across the UK in any day, Post Office accepts a large amount of 
money from customers paying their road fund tax. All this money 
needs to be paid to DVLA. Therefore, Post Office has a back-office 
activity - carried out centrally -of summing all these amounts of 
money and paying DVLA. DVLA knows how much money it expects 
to receive in this way and checks the amount it expects against the 
amount calculated by Post Office. This cross-check is an example of 
reconciliation and supporting it and reflecting its outcomes are 
central to Horizon.” 


The counter included both hardware and software. Dr Worden 
described the hardware in the branches as “not always reliable” 
but considered there were sufficient measures built into Legacy 
Horizon to ensure that hardware failures and communication 
failures could not adversely affect branch accounts. 
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Because of the difference in how data was and is stored in Legacy 
Horizon and Horizon Online respectively, availability of the 
network was required for the latter but not the former. In Legacy 
Horizon, sufficient data was held persistently in the branches, such 
that a branch could continue to trade, and could support most 
business applications, even if the wide-area network was 
unavailable. Whenever the network became available again, 
Riposte data replication should have ensured that the required data 
became available to the back-office systems. The only applications 
which could not run in this way were those that required some 
immediate validation from a client organisation - for instance, 
withdrawing cash from a bank account. Horizon Online this was no 
longer the case as the persistent data was all stored remotely in the 
BRDB. This meant that without a working network, a branch could 
no longer trade. The network infrastructure (which is another way 
of saying the reliability of the connections to the branch) by 2010 
had made this viable. 


The role of the Agent layer was to manage communications and 
translate data between the representation used in the branches 
and the network on Riposte, and the representations used in the 
Host layer. It is the Host layer that takes data from external clients. 
This is explained in the main design document for Legacy Horizon: 


‘The systems at the Host Layer can provide permanent storage for 
information if required by the application’s business rules. The 
Host systems can accept data from external Clients, and translate a 
file-based view of this information into discrete transactions or 
“messages”. These are then passed to the Counters via the Agent 
and Correspondence Layers. Similarly, messages received from the 
Counters are translated back into a file-based view for transmission 
to the external Clients.’ 


Each of the different business applications in Legacy Horizon was 
typically tied only to the different Post Office client in respect of 
whose business it was concerned. For example, Camelot does not 
need to communicate with the DVLA. Transactions in respect of 
them may be transacted in the same basket, but this facility is 
separate to the applications that deal with Post Office/Camelot 
business and Post Office/DVLA. Accordingly, each different 
business application in Legacy Horizon can, in Dr Worden’s words 
“be regarded as a vertical 'slice' though the diagram and is largely 
independent of the other slices”. He categorised this as “another 
example of robustness through architecture” but again, it could be 
seen as a design element. 


Visual Basic, C and C++ programming languages were used in the 
Legacy Horizon system along with Oracle development tools. 
Riposte, the product developed and marketed by a company called 
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Escher (and which featured in many PEAKs) was used in Legacy 
Horizon too. 


The Horizon Online counter and BAL (Branch Access Layer) were 
developed in Java, while the agents and host modules remained in 
C and C++. C is a general purpose code, an imperative procedural 
language which was invented by the late Dennis Ritchie. He was 
also one of the co-creators of UNIX with his colleague Ken 
Thompson. Neither of these gentlemen are household names in 
society generally in the same way as, for example, Bill Gates of 
Microsoft or Steve Jobs of Apple, but are very well known in the 
computer software world. Both C and UNIX are extremely 
important software technologies which are the heart of a great 
many, if not almost all, software products used today. Together Mr 
Ritchie and Mr Thompson won the Turing Award in 1983, a prize 
with similar status to a Nobel Prize, which is awarded by the 
Association for Computing Machinery. All of Java, C and C++ are 
well known languages that are in widespread use. 


Horizon Online was not a completely new system; it built upon, and 
continued to use, certain elements of Legacy Horizon. The move 
away from Riposte and Visual Basic to Java brought in more 
modern development tools and enabled techniques such as object 
orientation to be used. Object orientation means that modules must 
be self-contained, and will communicate via pre-defined and 
documented interfaces. These interfaces are not dependent on the 
application’s physical implementation. Third party and commodity 
products (such as operating systems and device drivers) are simply 
bought in by Fujitsu. Mr Coyne stated that “Horizon is therefore 
ultimately a composition of many sub-contracted components”, 
which I consider to be a fair description. 


Applications are developed by suppliers (who supply generic 
software or products), third parties who supply applications that 
meet Horizon requirements, Fujitsu developers who develop 
software to meet specific goals, or the Post Office. If the latter, this 
is done by configuring standard components. As at the date of the 
expert reports for the Horizon Issues trial, the architecture is data 
driven such that the Post Office could introduce new clients 
without reference to Fujitsu. All of the different applications must 
however integrate with the other existing applications, as well as 
any other new applications that are added. The complexity of the 
system will therefore have increased, year on year, as further 
products and applications are added. Once an application has been 
developed, it must be tested and evaluated, and then the process of 
integration takes place. It will then be released (which means 
introduced into live) and maintained thereafter. 


The Horizon system includes an audit database. This is an accurate 
record of any activity which can affect the branch accounts. These 
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records are available to be consulted in the event of any 
discrepancy arising anywhere in Horizon (for instance, due to a bug 
in some other Horizon application, or some operational error in 
running a batch process, or a dispute about what data was entered 
at the counter) but the parties could not agree on whether and 
when this should be done. The audit database is a robustness 
measure of a secure kernel (SEK) which also involves redundant 
data storage (RDS). In Legacy Horizon, all data travelled from the 
counter through the software application at the branch, through 
Riposte data replication to the two campuses, then through the 
Audit Agent to the Audit Store. In Horizon Online, all data travelled 
from the counter through the software application at the branch, 
through communications hardware and software to the Branch 
Access Layer (BAL), into the BRDB and then nightly to the audit 
store. 


The Post Office currently has several hundred client organisations, 
which shows the diversity of services and products available in a 
branch. This also makes it obvious that, for most of these clients, 
the service provided through Post Office will be different in nature 
from the service provided by the Post Office to its other clients, so 
some unique software functionality must be provided within 
Horizon both in the branch (for the counter) and also the back 
office to support the activities for each particular client. This is a 
part of what makes Horizon such a large and complex system. 


Checks are built into accounting systems. These can be by way of 
checks on data entry, double entry bookkeeping checks in two or 
adjacent layers, consistency checks and defensive programming. 


As I have already identified in Judgment (No.6), Dr Worden 
grouped countermeasures including within the Horizon System 
with other characteristics that are not properly described as that, 
together with those (such as manual checking by SPMs) that are 
not properly described in that way. 


In a Post Office internal presentation document dated 7 December 
2009, the following passages appears in relation to “Horizon - 
Current State”. I consider it an apt description of Legacy Horizon: 


“CURRENT STATE 

Before discussing the future development of Horizon, lets remind 
ourselves of the 'system' we have today 

OVERVIEW 

Horizon is 13 years old and the design criteria/technology was 
different and the business needs were different - the demand for 
on-line working was low, the telecoms were expensive for on-line. 


Horizon was built to support off-line trading. 
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Very high security (e.g .. user ID/ access/ screens encryption) this 
was a requirement from Gov because Horizon was built to support 
benefits payments. 

Horizon is the 2nd most secure system in Europe. The MOD being 
#1! 

Horizon is a costly system. For example Horizon is a bespoke 
system that uses a different encryption. Link for instance is unable 
to decrypt the encryption we have, so we have to decrypt before 
sending! 

Horizon is also a system that is wrapped up in 'barbed wire' - 
making changes difficult and costly - test everything! 

As time has passed and more product have been developed Horizon 
has evolved - from a technical perspective essentially we have 
bolted things on the side - we undo the barbed wire stick a bit in 
then wrap everything back up. 

Design was optimised at the time to minimise costs (esp. network) - 
offline working.” 

(emphasis added) 


The following headline points emerge from this: 


1. Horizon was originally designed to support benefits payments to 
welfare claimants. 


2. Horizon was already old in 2009. Although there were some 
major changes made when it became Horizon Online from Legacy 
Horizon, Horizon Online still used major components of the existing 
system. It is now somewhat older than it was in 2009. 


3. It was originally built for off-line trading. 


4. It has a relational database at its heart, but very numerous other 
applications that have been bolted on to it over the years as the 
Post Office’s business has evolved. 


5. There are a very wide number of other computing companies 
involved in the evolution of this system, not only in terms of 
software. Oracle, Escher Group, ICL/Fujitsu, ATOS, Computacenter 
and many more. It is a bespoke system that uses different 
encryption to other systems, such as Link. 


6. The complexity of the different interfaces, as a result, is very 
high. 


There have also been a total of some 19,842 release notes (in 
relation to software changes) in the life of Horizon. This is 
consistent with each of these notes being a change to the Horizon 
system. 


Fujitsu is the entity that was responsible for Legacy Horizon 
(putting to one side any contractual nuances or differences 
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between the Post Office and Fujitsu that may have existed, which 
are no part of this litigation) and it remains the main partner with 
the Post Office for Horizon Online. However, other suppliers are 
also now involved. Atos provides first line support via the Service 
Desk; Computacenter supplies the hardware, and Verizon is 
responsible for networking (what might also be called the 
connections between the different parts of the system). The main 
element within Fujitsu that was referred to in the evidence was 
SSC, in particular 3™ line support, and “Development”. SSC also 
serves other Fujitsu customers. 


The Post Office Account - headed by Mr Parker - is responsible for 
developing and maintaining Horizon, deploying it to branches, 
Managing the operation of the system and supporting users. The 
organisation of the Post Office Account is shown by the following 
graphic: 
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After the project to introduce the Payments Benefit Card in 1999 
(the tri-partite project to which I have already referred) was 
abandoned, the Post Office and ICL moved to automate post offices 
and this was the introduction of what is now called Legacy Horizon. 
In August 2000 there was the Release of what is called the Horizon 
Core System, or Core System Release. This introduced Automated 
Payment Smart cards and APS/TPS reconciliation into branches. 
Not all branches received installation of Horizon in the same 
month; that would, I imagine, be impossible in practical terms 
given the number of branches in the Post Office network. Some 
dates are included in Judgment No.3 from the Common Issues trial, 
in relation to those claimants called as witnesses in that other trial. 
Horizon was installed in Mr Bates’ branch in October 2000; Mrs 
Stubbs said it was installed in her branch “in the course of 2001” 
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but it must have been earlier than that, as letters in relation to her 
shortfalls both to, and from, Mr Manning are dated 1 and 2 
November 2000. The precise date of introduction of Horizon at 
each of the branches in the Post Office network is not relevant. 


An upgrade of Horizon was performed in February 2001, which is 
called Maintenance Release M1. The prime purposes of the 
upgrade were the enhancement of the CSR+ Applications (APS, 
LFS, EPOSS, EPS, OBCS), enabling of the AP client variable day 
file transmission, enhancement to Reference Data products and 
minor changes to TIPAIS Pathway generated CPs to improve 
operability of the system. 


A summary of some of the Releases appears in the chronology 
milestones for Legacy Horizon below at [96] in the list of 
chronology milestones. 


Horizon included what is called an Electronic Point of Sale System 
or EPOSS, which refers to the part of Horizon within the branch. 
The parties referred to this as activity at the counter, or simply 
“counter”. This term comes from the counter in the branch where a 
customer is served. In addition to this part of the system, there was 
another element which the parties (for shorthand purposes in a 
sense) referred to as the “back office” function. This refers to the 
way in which the Post Office reconciles its transactions with its own 
clients. 


EPOSS must allow the counter staff to record that some goods have 
been provided to a customer, compute the price of those goods, 
and allow the customer to pay the money required for all their 
purchased goods, for instance by cash or a credit card. If a 
customer wants to carry out two or more different activities in one 
visit to the counter - for instance, to settle a bill and to buy some 
stamps - Horizon does not require the customer to settle the 
amount in two separate pieces. Horizon has the concept of a 
customer carrying out a 'basket' of activities and settling the total 
amount due for the basket in several ways - by one credit card 
transaction, by a cheque, by cash, or by a mixture of these. 


At the stage when Horizon was introduced to branches - and 
indeed, this was a fundamental distinguishing feature of Legacy 
Horizon, compared to Horizon Online - the data was held in 
branches on terminals. Escher Group, a separate company, 
provided the Riposte Message Server (which was responsible for 
storing all the data in the Post Office branches and replicating it to 
the Data Centres). Terminals had two discs within them, one of 
which was termed a “mirror disc”, which retained all the data of 
what occurred on that terminal. The branch would have a hard 
disc, which would store the data for the branch and this would also 
be replicated on other counter positions (if a multi-counter branch) 
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or, in the instance of a single counter branch, stored to additional 
external storage on the counter. It would then be passed on from 
the counter to the Horizon data centre where it is stored in the CS 
messagestore. 


In the summer of 2000, a ‘proof of concept’ was undertaken to 
investigate the integration of internet technologies within the then- 
current Horizon System (which was being/about to be rolled out) to 
support the delivery of banking transactions. This was called the 
Network Banking Solution. A primary facet of the Network Banking 
Solution was the delivery of the banking transactions within the 
already established Escher WebRiposte environment. The full 
installation and integration of this was the task of Fujitsu (at the 
time ICL Pathway). WebRiposte was a message passing technology 
from Escher Group that extended the functionality of Riposte 
Message. 


IBM were selected for the supply of the Network Banking Engine 
which was designed to handle the interface between the Horizon 
System and the agreed Financial Institutions; these are also called 
Post Office’s clients, although in some documents they are referred 
to as ‘External Clients’. This was described by Mr Coyne in the 
following way: “This communication was initially based upon 
Escher’s Riposte messaging system. This was later supported by 
the WebRiposte system to accommodate Network Banking changes 
and the Network Banking Engine supplied by IBM to deal with the 
increase and change in type of business transacted in the branches. 
This was subsequently migrated to Horizon Online. Horizon is 
therefore ultimately a composition of many _ sub-contracted 
components.” 


It can therefore be seen that Horizon Online, although it includes 
some major differences (for example where the data is stored; this 
is no longer in the branches as with Legacy Horizon) built upon and 
adapted the existing functionality of Legacy Horizon. It was not an 
entirely new system. 


Mr Coyne opines that the introduction of network banking (as well 
as developments in the EPOSS function) brought with it a more 
complex enhanced architecture within which further systems to 
ensure transaction integrity and reconciliation could be imposed. 
However, he also considered that Horizon “originally stemmed 
from an inherited system and architecture with an_ initial, 
fundamentally different design requirement” (namely the Pathway 
Benefits Payment Card project). Regardless of that, there is no 
doubt that Horizon is a highly complex and multi-faceted system, 
because it has so many different elements and interacts with so 
many other systems. It has been added to multiple times over the 
years, as clients are added, new products are introduced, and 
indeed other sub-contracted parties and software products have 
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become involved. I do not consider that resolution of the Horizon 
Issues requires an analysis of the reason or reasons why, in 
historical terms, Horizon may have developed (or had inherent 
within it) any bugs, errors or defects. 


The Horizon System that was initially installed in the branches in 
2000 onwards is Legacy Horizon. ICL Pathway was (at least partly) 
owned by Fujitsu and I use the term Fujitsu to refer to the 
computer specialists with whom Post Office had contractual 
relations in respect of Horizon. I therefore make no distinction 
between the years when ICL Pathway or Fujitsu was or were 
contractually responsible for Legacy Horizon, or indeed with any of 
the other computer specialist companies such as Escher Group who 
provided Riposte. Legacy Horizon involved the installation and use 
of Horizon terminals within branches, which were used by the staff 
there (including the SPM) to transact all of the branch’s Post Office 
business with the branch’s customers. If a customer in a branch 
wanted to purchase something from the associated retail outlet, 
they would transact that business at a separate till, not at the 
Horizon terminal. The National Lottery is an exception to this, as 
certainly now, it spans both parts of the business of a branch. 


Mr Coyne recited some “chronology milestones” for Legacy 
Horizon, which he explained were not exhaustive but which he 
considered were milestone dates. Dr Worden also drew attention to 
changes during the period 2000 to 2010 which he described as 
significant changes. 


This chronology included the following: 


1. Pathway (later Fujitsu Services) awarded contract for Benefits 
Payment Card May 1996 


2. Horizon Pilot 1996 


3. Pathway cited “greater than expected complexity” and “...major 
implications for the degree of difficulty of the project” which 
ultimately lead to failure of the project. 


4. Post Office Counters Ltd and Pathway sign agreement to utilise 
the project to automate Post Offices July 1999 


5. Horizon rollout 1999 - 2002 


6. Core System Release - This included the introduction of 
Automated Payment Smart cards and APS/TPS reconciliation. 
August 2000 


7. Maintenance Release M1 - Prime purposes of the upgrade were 
the enhancement of the CSR+ Applications (APS, LFS, EPOSS, 
EPS, OBCS), enabling of the AP client variable day file 


transmission, enhancement to Reference Data products and minor 
changes to TIPAIS Pathway generated CPs to improve operability 
of the system February 2001 


8. S04 Release Additional functionality on the Horizon Pilot outlets 
to permit the printing of forms approx. July 2001 


9. S06 Release Day D rectification measures - this included a new 
automatically generated broadcast message to detail when 
counters at an outlet were offline. This was to be implemented in a 
staged manner and included a receipts and payments fix June 2001 


10. S10 Release Data centre and counter upgrade introduced 
unattended reboot facility at the counter September 2001 


11. B11 Release of the first network banking release, changed the 
version of Tivoli used by the whole estate in approx. December 
2001 


12. S11 Release January 2002 

13. B12 Release June 2002 

14. S20 Release September 2002 

15. B13 Release in approx. September 2002 


16. In 2003 there were different changes introduced. Network 
Banking was introduced at this time, as were the Data 
Reconciliation Services (DRS) and debit card processing. There 
were also the S30 and S50 releases which immediately follow this 
entry. 


17. S30 Mails Application /Escher Mails 3.3 package (1 Feb 2003) 
19. S50 Release October 2003 

20. S60 Release Approx. February 2004 

21. S52 Release March 2004 

22. S70 Release October 2004 


23. S75 Release (containing changes to support the changeover to 
use of NBX banking agents in approx. October 2004 


24. IMPACT Programme 
25. POLFS (a SAP-IS Retail System) was implemented in 2004 
26. S80 Release Jan 2004 - Aug 2005 


27. BI3 S80 T&T Harvester Agent accommodation was introduced. 
Mr Coyne thought that this may have been approximately Nov 
2004. 


28. POLSAP rationalisation (rationalisation of disparate systems 
SAPADS and POLFS) between 2007-2009 


28. Pension & Allowance Order Books were replaced by the Post 
Office Card Account in 2005. The Post Office had always had 
business relationships with banks such as Girobank. 


29. S90 ‘Bureau Plastic’ accommodation introduced in January 
2006 


30. S92 Release March 2006 

31.T10 Release August 2006 

32. T40 Release January 2007 

32. AP/ADC was introduced in around 2007/2008. 

In around 2010, POLFS and SAP ADS were merged to make 


POLSAP. 
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Horizon was migrated to Horizon Online in 2010. Horizon Online is 
also called HNG-X, but now it is on a different platform since its 
move to Windows 10 and it is now called HNG-A. The letters “NG” 
stand for “New Generation”. In 2010 there was no sudden change 
in the range of business applications supported by Old Horizon in 
the branches. This range of applications has increased continually 
over the lifetime of Legacy Horizon and Horizon Online. The main 
reason or rationale for the change to online was to exploit advances 
in the underlying communication technology, and improvements in 
its reliability. This meant that it had become possible to store all 
persistent data at a centre rather than in the branch (on the 
counters), with the consequence that a branch could only operate 
when communications were available, but the risk of failed 
communications was significantly lower in 2010 than it had been 
10 to 12 years before. Dr Worden considered that this change 
mirrored the wider changes across the IT industry, where 
increased reliability of communications infrastructure meant that 
applications could be dealt with in this way. 


The centralised storage of transaction data allowed the 
architecture to become simpler in some respects, with simpler 
management of the branches in the event of hardware failures or 
replacements and other events. This is because in those cases 
branch data would not be lost. There was also no dependence on 
Riposte data replication, which meant that Riposte could be 
removed entirely. Riposte is therefore not used in Horizon Online. 


One feature of Horizon which emerges from all the evidence 
provided is that it had a difficult interface with the systems of the 
Post Office’s clients, such as Camelot. It is not necessary to go into 
the technicalities of why that was, and was recognised by the Post 
Office (in this instance by Ms Van Den Bogerd herself) so far as 
Camelot was concerned by the introduction of the Ping fix in 2012. 


Horizon Online 
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A document entitled Counter Business Architecture document of 4 
August 2017 stated that “The objective of the HNG-X programme is 
to develop a system with structural and operational characteristics 
that substantially reduce ongoing support and maintenance costs 
with respect to the current Horizon system” and also “The overall 
requirement is that the business capabilities offered by the current 
system (Horizon) are preserved in the new system (HNG-X). 
However, a limited number of business capabilities will be revised 
based on a joint optimisation of business requirements and system 
properties.” 


The fundamental change was that in Horizon Online, no transaction 
data was held in any persistent form in the branches. The same 
document explained that storage of transactional data within 
counters in the branches led to the need for security mechanisms 
that affected “both the structural complexity and operational 
performance of the Counter Business Application.” 


In Horizon Online, before completion of a basket of customer 
services, that basket was transmitted and had to be acknowledged 
by the BRDB and the basket could not complete successfully at the 
counter until that had happened. This is why Horizon Online could 
not operate in the branches without working communications. The 
main difference at the “back-end” was the existence of the BRDB, 
which was the main persistent store of all transactions for all 
branches. Many business applications in the back-end were 
unchanged (and were referred to as 'legacy'), except for the need 
for them to interface with the BRDB rather than with the previous 
Agent layer. Other copies of transaction data continued to be 
stored in those applications. 


The previous branch architecture had been based on Riposte, 
which provided functionality on different levels. These included 
user interfaces, some business applications, and message storage 
and replication. Because in Horizon Online, Riposte was completely 
removed, all elements of the branch software were replaced. A 
great deal of the Legacy Horizon branch code had been written in 
Visual Basic, for Horizon Online nearly all the branch software was 
written in Java. Java is a newer language, and supported more 
modern programs (or what Dr Worden called modern programming 
paradigms). This allowed a more modern software architecture in 
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the branch, which did not have to be fitted around the existing 
architecture of Riposte. 


The new architecture is shown in the following diagram: 
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The top 'Presentation' layer is the one responsible for displaying 
information to the user and for accepting the inputs. The next 
'Interaction' layer provides the foundation or building blocks for 
this interaction, such as menus. The effect of these two layers was 
to provide a user interface similar in style to that which had been 
provided by Riposte in Old Horizon. This therefore meant that the 
experience of the person, either SPM or assistant, using Horizon 
would be similar to what it had been in Legacy Horizon, but the 
system would be using Java technology, rather than Riposte as 
before. 


The 'Business' layer provides the functionality of the different 
business applications, which is done in what is called an object- 
oriented fashion. This means that there are several general- 
purpose software objects (i.e. modular blocks of software) with 
names such as customer, basket and payment, which represent the 
behaviour that is required of those in the real world, but is done in 
a way that can be easily reused in many different business 
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applications. There are therefore certain core design elements used 
for many different applications. 


Business process objects and business data objects are more 
specialised to support the many business applications. Business 
process objects support the sequence of steps which make up a 
business process, and the business data objects hold the necessary 
data, which is presented at the counter or stored. As has already 
been noted, this is generally not done by writing completely 
different software for each business application, and a great many 
applications are driven by reference data. This data defines the 
sequence of steps in completing each type of service for a 
customer. This reference data-driven style of software is common 
modern practice. The theory is that new applications can be 
supported just by adding new reference data, rather than by 
writing new software. There are different capabilities in the 
business layer which it is not necessary to list. 


The Services layer provides a set of software objects which provide 
services in support of many business applications. Most of these 
services deal with organising information and sending it for storage 
in the BRDB. The facilities for ensuring that each basket is zero- 
sum before it is sent, and transactional integrity are provided 
generically in the services layer. These functions do not therefore 
need to be coded individually in the business objects, and do not 
have to be built individually into any new business application. The 
facilities for zero-sum baskets, which is one of the ways of 
complying with the requirement for double entry bookkeeping, and 
transactional integrity are provided generically in the services 
layer. 


The Process Engine is a key component of the Services layer, and 
provides a simplified way for the counter to provide services which 
involve a sequence of steps. The sequences of steps need not be 
defined in Java code but are defined in a specialised Process 
Definition Language (PDL), which is executed by the Process 
Engine. PDL was developed for Horizon Online by Fujitsu. The use 
of PDL means that complex sequences of steps are simpler to 
define and test. This is an example of generic data-driven software. 


Some data are stored persistently on the branch hardware (as 
shown by the disc-shaped boxes in the diagram) however, these 
data do not include customer transaction information. These 
include business process definitions (definitions of sequences of 
steps in a process), other reference data, data defining reports that 
can be output in a branch, and other information required to 
support operations. The reference data is refreshed daily from the 
data centre. There are services which provide these data to the 
other layers in forms that are convenient for them to use. 
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. The Services layer of the branch architecture is the only layer 


which communicates with the data centre, through the 
communications subsystem. The purpose of this layered approach 
is to provide each kind of functionality (such as reliable and robust 
communication with the data centre) in one layer only, and not 
have to reinvent it for many different business applications. In 
effect, the services layer in Horizon Online now provides many of 
the services which were formerly provided by Riposte. 


The Services layer also provides interfaces for online services, 
where it is necessary to contact another non-Post Office IT system 
in order to provide the service to a branch customer. These online 
services include banking, the use of credit/debit cards, mobile 
phone top-ups and other services. 


Both the Branch Access Layer (BAL) and the BRDB are completely 
new elements of the Horizon Online back-end. The principal 
function of the BAL is to exchange messages with the counter 
software in the branches. It also has a role in checking that the 
information is conformant, logging of all exchanges, and recovery 
from many kinds of error conditions. Because it has to handle more 
than 25 million transactions per day, the BAL has many design 
features to ensure high performance (principally by distributing the 
load in parallel across many machines), as well as robustness - for 
instance, through reliable and redundant hardware. 


The branch database or BRDB is a large, high-performance Oracle 
database whose main function is to store all customer transactions 
which originate in any branch. It also has features to ensure high 
performance and robustness. These are, for example, through the 
use of transactional integrity and recovery. 


There are other notable milestones in my judgment of the Horizon 
System which are not included in the list above at [96] include the 
following: 


1. Migration to Horizon Online in 2010. 


2. The discovery of what has become called the Callendar Square 
bug. This was experienced and reported in that branch in 2005. Mr 
Godeseth’s evidence was that it had probably been in the Horizon 
System since 2000. Ms Chambers’ entry in one particular PEAK in 
February 2006 was that “this problem has been around for years 
and affects a number of sites most weeks.” The correct place to put 
it in any chronology, on the basis of the evidence currently before 
the court, is 2000. 


3. The introduction of what was called the Ping fix, which was in 
relation to the accounting with Camelot for the National Lottery, 
which was initiated in 2011 and occurred in 2012. Much evidence 
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was given by Ms Van Den Bogerd, both in the Common Issues trial 
and the Horizon Issues trial, about the improvement in accuracy of 
accounting regarding Lottery sales since the Ping fix was 
introduced. 


4. In June 2014 ATOS started to provide 1* line support. The 
Management and operations with regard to Reference Data has 
been outsourced from within Post Office control to ATOS since the 
same date. 


5. The move from HNG-X to HNG-A (which uses Windows 10 rather 
than Windows NT4). This change to Windows 10 occurred in 
February 2017 and the roll out was administered by 
Computacentre. 


Having provided what is essentially an overview of the Horizon 
System, both Legacy and Online, it is convenient to turn to the 
main battleground between the parties concerning the subject 
matter of the Horizon Issues trial. This is the presence of bugs, 
errors and defects within both Legacy Horizon, and also Horizon 
Online. This can be done most conveniently by retaining the 
numbering of the different alleged bugs from what was called “The 
Bug Table” in the 2™ Experts’ Joint Statement. 


Submissions and Evidence 


There were two features of the Horizon Issues trial which were 
somewhat unusual. This was a lack of detailed evidence adduced by 
the Post Office in relation to the specific factual matters recorded 
in PEAKs and KELs in particular, and the way that this situation 
was sought to be remedied. This second element was by way of 
points explained or put “on instruction”, and also in submissions. 
This is referred to in [69] to [71] of the judgment that accompanies 
this appendix. It is an obvious point that submissions are not 
evidence, and vice versa. 


The first of these, the putting of points “on instruction”, in this case 
related predominantly to the way that certain matters were put to 
Mr Coyne in cross-examination to correct evidence of fact that had 
emerged during the cross-examination of Mr Godeseth. The second 
arose in the context of the Post Office’s Appendix 2 to its written 
Closing Submissions. The need for both, or either, arose in my 
judgment because of the way Dr Worden approached his task in 
comparison with Mr Coyne. Because he had done what he himself 
described as a “high level” analysis of Horizon, his written reports 
did not deal with the detailed contents of so many PEAKs and KELs 
in the same way that Mr Coyne did. Therefore, when the Post 
Office came to challenge Mr Coyne’s conclusions in closing 
submissions, they could not simply meet his evidence with 
opposing expert evidence of the same type. 
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Lengthy written opening and closing submissions were provided by 
both parties, as is usually the case in modern and complicated 
cases. The Post Office’s Closing Submissions in particular were 
very lengthy, and included appendices, one of which, Appendix 2 at 
Section K2, itself ran to some 137 pages. This document attempted 
to collate material relevant to the Bug Table, in a logical form, 
dealing with each bug in numerical order. It identified key 
documents, expert evidence (usually from Mr Coyne), entries in 
joint statements by the experts, and other sections including 
“relevant discussions during trial.” 


During the oral closing submissions, the claimants identified 
certain instances in which positive evidential assertions were made 
in Appendix 2 that were not, it was said, contained in the evidence. 
Particularly given Appendix 2 was so large, it was simply not 
possible to resolve these objections during that oral hearing. I 
therefore directed that the claimants serve a schedule identifying 
all such passages to which objection was taken, so that a response 
could be provided by the Post Office. This led to a “clarification” 
schedule served by the Post Office, which stated in respect of each 
challenged passage the following: 


Whether the passage was based on evidence in the trial, and if so 
the reference; any relevant documentary evidence; if documentary, 
whether that document was deployed in the trial and if so when; 
and/or whether the Post Office relied upon what Mr De Garr 
Robinson submitted would be a suitable response in some cases, 
namely common sense. 


. That clarification document was itself rather lengthy. It contained a 


further 36 pages, with 175 different entries. Of those entries, 115 
were to documents, and in only 28 cases had those documents been 
referred to at the trial. Further, some of the submissions went 
rather further that the documents supporting them justified. For 
example, on 8 Recovery Issues in the Bug Table (which was not 
accepted as a bug) the Post Office has submitted the following in 
relation to a necessary software fix: 


“It went to the Model Office on 1 June 2010 and Fujitsu have 
advised that it would have been rolled out to the rest of the HNG-X 
pilot by mid-June.” 


The supporting reference was to a document, PEAK PC0199000. 
However, the submission “Fujitsu have advised that it would have 
been rolled out....” reads as though Fujitsu have provided evidence, 
behind the scenes, and not called at the trial, about what would 
have or did happen. Further, the supporting entry in the PEAK 
relied on for the entries for 1 June 2010 and throughout that month 
are as follows: 
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“Date:01-Jun-2010 14:05:40 User:John Budworth 

Pre-Requisite Reference Data and Data Centre BAL upgrades have 
been applied to live. 

Please one-shot to the distribute and commit to Model Office 
branch 699010 by 15:30 today. 

Please distribute tonight to the 50 branches in Pilot 50 
RNT9566.xls attached as evidence. 


Date:04-Jun-2010 17:20:16 User:John Budworth 
Committed to Pilot 50 RNT9566.xls by MSS on Thursday June 3rd 
as verbally requested by RM. 


Date:14-Jun-2010 11:57:50 User:John Budworth 
Please distribute and commit to the balance of the HNGX estate 


Date:14-Jun-2010 11:58:11 User:John Budworth 
Evidence File Updated - RNT9566.” 


This entry reinforces my view that the submission goes a little 
wider than the document’s contents suggests. However, this is a 
rare example, and in any event I am entitled to draw an inference 
(that is, a common sense conclusion) that if the PEAK requests 
distribution to the entire estate, that did occur. For the most part 
this type of example is rare, and it is not usually the case that the 
submissions go further than the underlying documents suggest. I 
would like to record that I found Appendix 2 very useful; it must 
have taken an enormous time to produce. There are some 
omissions within it; for example, Dr Worden’s cross-examination is 
not always referred to. However, the task of producing this 
appendix to the judgment has been rather easier as a result of it. 


In order to use it however, it was not necessary to make a ruling on 
each and every one of the 115 different passages challenged by the 
claimants and the responses to those challenges by the Post Office. 
I have only referred to the actual material necessary in order for 
me to come to findings on the Bug Table, although I have 
considered the full content of Appendix 2 and its accompanying 
clarification document. I have not given documents that were not 
referred to at all in the trial the same weight as those that were, 
and I have not given those that were, the same weight as the actual 
evidence that was deployed in the trial. I have however considered 
them all. 


Bugs, errors and defects and their symptoms 


The first three of the Horizon Issues have the heading “accuracy 
and integrity of data”. Given the purpose of the Horizon System, 
both Legacy and Online, and the fact that it was the accounting 
mechanism by which branch accounts were produced in terms of 
the Branch Trading Statement, the presence of bugs, errors and 
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defects with the potential to cause apparent or alleged 
discrepancies or shortfalls in branch accounts was an extremely 
important element of the Horizon Issues trial. This is plainly 
included within Horizon Issue 1. 


The degree to which the Branch Trading Statement accounts did or 
did not, in law, have the effect of settled accounts between 
principal and agent was addressed in Judgment (No.3) on the 
Common Issues. The majority of the Horizon Issues - and certainly 
the first three - were all aimed at the same principle, namely (in 
summary) that of computing and accounting accuracy of the 
Horizon System in both Legacy and Online form. It was doubtless 
for this reason that so much of the cross-examination of the two 
experts dealt with the presence of bugs, errors and defects. 


In my judgment, the most convenient approach is to consider each 
of the alleged bugs, errors and defects in the Experts’ so-called Bug 
Table in order to come to a judgment on the existence, or 
otherwise, of each. Horizon Issue 1 requires this, and in addition, 
Horizon Issue 3 should be answered in the context of whatever the 
answers are to the different component elements of Horizon Issue 
1. This also means Horizon Issue 3 can be considered in the 
knowledge of how many, if any, of the alleged bugs errors and 
defects in fact existed in Horizon over time, and what their 
potential effect was. By the end of the Horizon Issues trial, in 
reality there was less in dispute between the parties on the 
different bugs in the Bug Table than appeared at the beginning of 
that process. This is not to say that there were many concessions, 
certainly in terms of the existence of bugs, although there were 
some. However, the expert evidence of both Dr Worden and Mr 
Coyne, although they differed in their conclusions, was not so far 
apart on the factual existence of certain bugs, errors and defects. 
Given the use during the trial of so many PEAKs and KELs, which 
contained contemporaneous records at Fujitsu of the effect upon 
SPMs of different features within Horizon, this is not so surprising. 
Some overarching themes remained, for example the difference 
between transient and lasting impact of bugs. These are addressed 
in the body of the judgment itself. 


Turning to the 29 different entries in the Bug Table in turn, and in 
summary terms only (as to reproduce references to all of the 
documentation, evidence and submissions on each would lead to a 
very lengthy analysis of each such bug, which in my judgment is 
disproportionate and not necessary), my findings on these are as 
follows. Because I am using the same numbering as in the bug 
table, the following bugs are not dealt with in this appendix in 
chronological order, but in the order chosen by the experts in their 
joint statement. 


1. Receipts and Payments Mis-match bug. 
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131. 
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This bug occurred in 2010. It is a bug which appeared in Horizon 
Online. It was agreed in the Bug Table as an “acknowledged bug” 
which had an impact on branch accounts. Dr Worden added in his 
comments that “Therefore, the extent of this bug is well 
established, in the GJ analysis.” GJ is Gareth Jenkins. It is accepted 
by the Post Office in paragraph 7 of Appendix 2 as one of a number 
of “bugs with lasting impact (although they were resolved)”. In the 
summary in Appendix 2, the Post Office stated that “In the event, 
however, this bug resulted in transient impact only.” I have dealt 
with the concept of transient impact in the main judgment on the 
Horizon Issues. 


The effect of this bug was identified upon the accounts of 
approximately 60 branches. Dr Worden and Mr Coyne, in the 
agreed entry in the 2”! Joint Statement, had stated that “the 
number of distinct bugs, for which the experts have seen strong 


evidence of the bug causing a lasting discrepancy in branch 
accounts, is between 12 and 29.” (emphasis added) 


Dr Worden had therefore seen “strong evidence of lasting impact” 
in respect of this bug. This was one of his 12. The same joint 
statement said it had affected “approximately 60 branches”. Mr 
Coyne in his report used the more precise number of 62. Of these 
62, two branches were affected twice. 


The bug related to the process of moving discrepancies into the 
local suspense account. The majority of incidents are recorded as 
occurring between August and October 2010. The bug was 
documented in a report from Mr Gareth Jenkins dated 29 
September 2010 where it was stated: 


“This has the following consequences: There will be a receipts and 
payment mismatch corresponding to the value of discrepancies that 
were lost. Note that if the user doesn't check their final balance 
report carefully they may be unaware of the issue since there is no 
explicit message when a receipts and payment mismatch is found 
on the final balance (the user is only prompted when one is just 
detected during a trial balance)” 

(emphasis added) 


This issue is reported as causing discrepancies that disappeared at 
the counter or terminal when the branches followed certain steps, 
but which persisted or remained within the back-end branch 
account. It is therefore something which is contrary to the principle 
of double entry bookkeeping, and should plainly not have occurred. 
The issue occurred when a branch cancelled the completion of the 
trading period and then, within the same session, rolled into a new 
balance or trading period. Because the discrepancy disappeared at 
the counter, the SPM would not know that the discrepancy existed. 
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The lack of an explicit message to the user, in my judgment, simply 
compounds the problem. This bug was acknowledged by the Post 
Office in its letter of response to the letter of claim sent in relation 
to the litigation. That letter of response was dated 28 July 2016. I 
am unaware of any acceptance by the Post Office of this bug prior 
to that date, but in any event that is not relevant to the substantive 
issue of answering the Horizon Issues. 


The Post Office in its Closing Submissions stated that this bug only 
occurred if there was “an unusual sequence of events”, namely as 
follows. If a branch had an unresolved discrepancy, when balancing 
the stock unit, and if the SPM pressed the Preview or Print button 
to produce the Trial Balance Report, this would cause the counter 
to return to the rollover screen. Having checked the Trial Balance 
Report, if the SPM then pressed the rollover button, Horizon would 
then ask the SPM whether they would like to transfer the 
discrepancy to the local suspense account or cancel the rollover. 


If they chose to cancel the rollover, this would cause the system to 
return to the rollover screen, allowing the SPM various choices, 
including cancelling the rollover process or the choice of again 
pressing the Preview or Print button to produce the Trial Balance 
Report. If the SPM then proceeded to rollover to a new Trading 
Period in the same session the fault would occur. This was a bug in 
the code that was triggered by the ‘cancel’ button being pressed 
and this incorrectly caused the discrepancy to be cleared so far as 
the user in the branch was concerned. 


I do not accept that this was a particularly unusual sequence. If 
there was a discrepancy, I do not see why it is suggested it would 
be unusual for a SPM to decide to cancel the rollover. This is, in my 
judgment, something which could be expected to happen 
sometimes, and plainly the absence of any message to the user 
could potentially contribute to this. 


The Post Office also, in the main body of its submissions, stated 
that Fujitsu monitors for receipts and payment mismatches, and 
investigates such occurrences. However, that does not mean that 
this bug does not exist; rather to the contrary. The fact that Fujitsu 
was supposed to watch for it, and then investigate it, does not 
affect its existence. Further, the submissions also relied upon the 
fact that instances of this bug “were collected through [Fujitsu] 
reports without the need for an SPM to report an issue.” In my 
judgment this overstates the factual position in the Post Office’s 
favour. PEAK PC0204263 dated 16 September 2010 shows that this 
PEAK was raised by a “customer call” on 13 September 2010, 
which means a SPM, who had a discrepancy of £109. This was 
given call priority B by Fujitsu. The SPM did report the 
discrepancy; they could not report a payments and receipts 


mismatch because they would not have known either about the 
existence of the bug, or its effects. 


138. That this was a problem that required a code fix is shown by the 
following entry in the PEAK dated 24 September 2010. 


“HNGX CODE FIX 


FIX DESCRIPTION 

Described Above 

PROPOSED BRANCH 

TBD 

COUNTER JAVA FILES CHANGED 
None. 


COUNTER PDL FILES CHANGED 
RolloverStockUnitBLO.pdl updated 
COUNTER REFDATA FILES CHANGED 
None. 


SHARED CODE FILES CHANGED 
None. 


BAL JAVA CODE FILES CHANGED 
None. 


SQL FILES CHANGED 
None. 


OTHER FILES CHANGED 
None. 


APPROPRIATE CODE COMMENTS 
YES 


DEPENDENCIES 
None. 


RELATED PROBLEMS 
None 

UNIT TESTING EVIDENCE 
screenshots attached. 


REGRESSION TEST CLASS 
NA. 


BACKWARDS COMPATIBILITY 
its a reference data change hence backward compatibility test is 
required on LSt 


139. 


140. 


DEVELOPMENT DOCUMENTATION 
None. 


REQUIREMENTS DOCUMENTATION 
None. 

HELP 

None.” 


The Counter PDL files were changed, and as this was a reference 
data change, certain checks were needed which is called a 
backwards compatibility test. This is to ensure that the fix itself 
does not cause other problems. This led, eventually on 5 October 
2010, to the entry in the PEAK of “Product Error Fixed” after a 
patch had been released. A patch is a change, or set of changes, to 
a programme or its supporting data, and it is something that 
changes the operation of the software. The entry of 14 October 
2010 states: 


“Ths Reference Data hot fix for this PEAK has completed LST 
testing and has been passed to the Reference Data Team for live 
deployment. 


Note this hot fix will only work when a counter receives the 02.12 
counter upgrade that is currently in live pilot. (Overnight package 
COUNTER X0212 56 1 or immediate COUNTER APP 56 1). 


[End of Response] 


Response code to call type L as Category 71 -- Final -- Fix Released 
to Call Logger Routing to Call Logger following Final Progress 
update. 


Service Response was delivered to Consumer.” 


I find that this is a bug with potential lasting impact, and indeed it 
did cause actual impact. The fact that the Post Office, once it was 
discovered that it was a bug, corrected the accounts affected by 
issuing TCs (and in one instance Fujitsu injected the necessary 
correction into the branch accounts) does not reduce its 
importance. A reference data fix was necessary to correct it. As 
explained above, reference data was part of the architecture in 
Horizon, and in my judgment is part of the software. The fact that 
aspects of the system such as the SQL files or the BAL Java code 
was not changed does not mean that this was not a software defect. 
Even though the underlying code was not re-written or corrected 
does not minimise the impact of this bug which required a change 
to the reference data as shown in the PEAK above, and other of the 
contemporaneous Fujitsu documents. 


2. Callendar Square/Falkirk bug. 
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It is agreed that this bug occurred between the years of 2000 and 
2006, although there is an issue about when it stopped which I deal 
with at [149] below. It is a bug which was present in Legacy 
Horizon. It too was agreed by the Post Office in the letter of 
response in the litigation of 28 July 2016. It was also agreed in the 
Bug Table as an “acknowledged bug” which had an impact on 
branch accounts. It is accepted as a bug by the Post Office in 
paragraph 6 of Appendix 2, but is said to have had “transient 
impact”. Dr Worden added different comments, including that “the 
bug arose from a fault in the underlying Riposte software, so it is 
not surprising that it took Fujitsu some time to understand it, or 
that they had to rely on the suppliers to fix it. It does not show poor 
system design or support by Fujitsu”. That latter sentence is in my 
judgment a little surprising. It is not relevant to the Horizon Issues 
whether fault or defects can be laid at the door of Fujitsu, Escher 
Group (who supplied Riposte) or any of the other specialist 
companies who provided software for different parts of Horizon. Dr 
Worden seemed to me to be very defensive on Fujitsu’s behalf. The 
Horizon Issues are about the operability and functionality of 
Horizon, and there are no findings made in the judgment or this 
Technical Appendix regarding which specialist company is to blame 
for the presence of any bugs, errors or defects. 


Riposte software was part of Legacy Horizon, although it is not 
used in Horizon Online, as one of the main changes between the 
two different types of system (Legacy, and Online) was specifically 
the way data was stored and messages sent. Legacy Horizon used 
Riposte and it was a central feature of that system. Riposte is not 
used in Horizon Online. The fact that Riposte is a product designed 
and supplied by Escher, and not Fujitsu, is not relevant to the 
dispute between the Post Office and the claimants. In any event, 
and as accepted and recited in paragraph 2 of the Post Office’s 
Appendix 2 to its closing submissions dealing with this bug, “Mr 
Coyne asserts that Bug 2: Callendar Square is a bug with lasting 
financial impact and in JS2, Dr Worden appears to agree that there 
is strong evidence of this....” 


In my judgment Dr Worden does more than “appear to agree”. 
Given the entry in the experts’ joint statement, he does in fact 
agree. This bug occurred when a SPM was transferring cash 
between different “stock units”, which means positions. 


One of the Post Office’s submissions on this accepted bug merits 
quotation in full. The Post Office in its Appendix 2 stated: 


“Fujitsu was aware of the wider Ripsote issue for quite some time. 
Its various manifestations were being identified and resolved long 
before the Callendar Square PEAK was raised.” 
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This is undoubtedly correct; Fujitsu was aware of the bug “for quite 
some time”. I do not consider that this assists the Post Office’s 
position on the Horizon Issues. The knowledge Fujitsu had about 
issues with Riposte was not shared with SPMs, so far as I can tell. 
This is also the bug that a Fujitsu employee, Anne Chambers, 
stated in an email in February 2006: 


“Haven't looked at the recent evidence, but I know in the past this 
site had hit this Riposte lock problem 2 or 3 times within a few 
weeks. This problem has been around for years and affects a 
number of sites most weeks, and finally Escher say they have done 
something about it.” 


The Post Office stated in its submissions that “Fujitsu’s automatic 
reports identified instances of the Callendar Square bug without 
the need for each SPM to call in with a report”. That submission 
ignores the numerous examples in the numerous PEAKs where 
SPMs did in fact call in with a report of a discrepancy. The Post 
Office also, entirely incorrectly in my judgment, complains in its 
Closing Submissions that it did not have the time to cross-examine 
Mr Coyne fully on this bug. It submitted the following: 


“Mr Coyne notes from the documents that the Callendar Square 
bug was “operating and resident in the system for years without 
any comprehensive linkage being observed by Fujitsu”. However, 
this provides only an incomplete picture, as Post Office would have 
demonstrated if it had had the time it would have needed to cross 
examine him on this bug (that time was taken by Mr Coyne’s 
extraordinary changes in evidence on the afternoon of Day 17, the 
final afternoon of his cross examination).” 


I reject this submission for two reasons. Firstly, it was a matter for 
the Post Office’s legal team how long it spent on each subject with 
Mr Coyne, who was cross-examined for four days. It might have 
been sensible to have started with the number of bugs for which 
Mr Coyne was contending, and then cross-examined on each or 
some in detail after that. The Post Office chose to leave the total 
number of bugs until the last day; there is nothing wrong with that, 
but it is not correct to state that there was insufficient time to 
cross-examine on this bug (or any bug) and try to blame Mr Coyne 
for it. Secondly, the submission entirely ignores the evidence of Mr 
Godeseth, called by the Post Office, who expressly accepted that 
the bug had probably been present since 2000. 


The Post Office also submitted that “a Peak would be raised and the 
SPM made good in the ordinary course” if this bug affected branch 
accounts. There are two points that this submission fails to 
address. Firstly, the Horizon Issues concern bugs, errors and 
defects with the potential to affect branch accounts. The Callendar 
Square bug is clearly one of these. Secondly, the “ordinary course” 


in this instance seems to be that not one of the SPMs across the 
Post Office branch network were told about the existence or effect 
of the Riposte lock problem, which Fujitsu recorded (by Anne 
Chambers in February 2006) had “been around for years”. In my 
judgment, the ordinary course ought to have been providing SPMs 
with notification of the existence of this bug and its effects. SPMs 
and the Post Office have agreements in place (at the time of this 
bug the SPMC) that govern their legal relationship. Judgment No.3 
makes clear what the scope of the parties’ legal obligations are. 


149. One specific factual point of disagreement between the parties was 
the period of time when the effects of this bug were no longer 
experienced. The Post Office maintains this came to an end in 2006, 
and the claimants submitted that it ran until Riposte was no longer 
used, namely 2010 when Horizon Online came into being. One 
passage of re-examination of Dr Worden is pertinent in this respect: 


“Q. You say another symptom of the Riposte problem. Perhaps you 
could explain a little bit what you mean by that? 


A. Well I did explain and I will explain again. That there is a 
problem in Riposte which leads to a failure to replicate -- failures to 
release a lock I think -- and then on certain occasions that leads to 
a short-term failure to replicate. On other occasions there is a so- 
called event storm during which there is a longer term failure to 
replicate and during these failures to replicate all sorts of different 
things may happen, for instance they double transfer into a stock 
(inaudible), a precise Callendar Square phenomenon; whereas 
other things such as system freezes, I can’t remember all the other 


details, but there are many other symptoms of then underlying 


Riposte problem and they have been noted over this whole period.” 
(emphasis added) 


150. By “this whole period” Dr Worden was, in my judgment, clearly 
referring to the period 2000 to 2010. Whether the short-term 
failure to replicate, and the “event storm” with its longer term 
failure to replicate, are seen as one bug in Riposte (which is the 
way the experts approached it) or two sub-elements of one bug, 
which it could be (given one is a short term failure, and the other a 
long term failure, both being failures to replicate) does not for 
these purposes matter. In my judgment, the period when the 
effects of this occurred are 2000 to 2010. Dr Worden’s evidence 
does not support the Post Office’s position that this ended in 2006, 
and is indeed to the contrary. 


151. I find that this is a bug with potential lasting impact, and indeed it 
did cause actual impact. 


3. Suspense Account bug. 
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This is a bug in Horizon Online and its identified years of effect are 
2010 to 2013. It was also agreed in the Bug Table as an 
“acknowledged bug”. This bug was the third of the existing bugs 
acknowledged by the Post Office in its Letter of Response on 28 
July 2016. 


It is accepted by the Post Office in paragraph 7 of Appendix 2 as 
one of a number of “bugs with lasting impact (although they were 
resolved)”. Dr Worden’s comments do not include that it had an 
agreed impact upon branch accounts. This is because of the 
concept he introduced (expanded in his reports) of “transient 
effect”. What this meant was that if there was an impact on branch 
accounts by something within Horizon, but that was then corrected 
by a TC, he concluded it had a “transient effect” upon branch 
accounts and dealt with it differently. I address this concept in the 
substantive judgment under “lasting and transient effect”. Other 
comments of his in relation to this acknowledged bug are: 


“It was a transient effect arising not from a fault in the software, 
but from a change in database archiving policy in 2010. The delay 
in correcting it arose from a failure of communication between PO 
and Fujitsu. Because the bug would only manifest itself annually for 
any affected branch, the effects of this delay were not widespread.” 


Peak PC0223870 shows that Fujitsu were able to identify the 
branches affected, even when Subpostmasters did not report it. 
There is evidence that the branches were compensated, as I would 
expect from the normal error correction processes.” 


The Post Office accepts in its Closing Submissions that Fujitsu 
identified affected branches by use of “historical data”, in other 
words, it had to research it as an investigation was required. Mr 
Gareth Jenkins in 2013 prepared a note entitled “Local Suspense 
Problem” which identified that, at that stage, 14 branches had been 
affected. He stated: 


“The root cause of the problem was that under some specific, rare 
circumstances some temporary data used in calculating the Local 
Suspense was not deleted when it should have been, and so was 
erroneously re-used a year later. When the SPMR was asked to 
clear Local Suspense the actual (ie incorrect) amount was recorded 
in the Audit Trail. This means that there was no corruption in the 
audit trail and it accurately reflects the transactions that occurred 
in the Branch. 


If the BTS from the previous period was taken to provide a set of 
Opening Balances and all transactions that were logged to the 
audit trail during the period were taken as adjustments, then this 
would show the correct value that should be in the Local Suspense 
account.” 
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He also stated, in a passage that is of interest in terms of the 
dispute between the parties about the use of Audit Data or 
management data such as Credence (the Post Office maintains that 
the latter are sufficient, the claimants insist the former are the 
relevant accurate records) the following: 


“As well as passing these Local Suspense transactions to the 
normal accounting tables that are used to update POL SAP and 
Credence, they are also written to a table in the Branch Database 
that is used to support the printing of the Branch Trading 
Statement (BTS) after that Branch has been fully Balanced.” 


This demonstrates, in my judgment, that in this instance Credence 
would have been wrong. Further detail of the problem explained by 
Mr Jenkins in the same report was: 


“In April 2011 a problem was found with the archiving strategy 
related to Stock Units that have been deleted in a Branch. A 
consequence of this is that some changes were made to the 
archiving strategy on 3 July 2011. An unintended consequence of 
this change was that any Branch that deleted a Stock Unit at the 
end of 2010 which had local suspense transaction in that Stock 
Unit before it was deleted were left in the table used for 
constructing the BTS. This meant that as Trading Periods cycle 
around each year, these BTS records became visible in 2011 when 
the same Trading Period was reached. 


The effect of these old records was that after the BTS was 
produced an incorrect figure was generated for the Opening 
Balance of the Local Suspense Account for the following period. 
This amount corresponded to the value of the historical record. 


These orphaned records were created between 16 November 
2010 and 9" December 2010.” 


This meant that the figures for the preceding year were effectively 
recycled as correct figures for the subsequent period. The Note 
prepared by Mr Jenkins makes it clear that, despite how the Post 
Office sought to present this in their submissions, this problem 
persisted. 


The document states: “This problem was not reported to Fujitsu in 
2011/12 and only affected a small number of Branches and only for 
a single Trading Period. However the two branches with the largest 
discrepancies did report the issue to Post Office Ltd who could see 


the impact of the problem in their back end system and wrote off 


the loss or gain for the branch but did not ask Fujitsu to investigate 
further. At the same Trading Period in 2012/13, the problem re- 


occurred and this time one of the affected Branches reported the 
problem to Fujitsu on 25" February 2013 (Peak 223870) resulting 
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in a detailed analysis of this issue and finding the orphaned BTS 
records. The root cause was determined by 28" February 2013 and 
a preliminary report was sent to Post Office Ltd. A further update 
was sent on 14" March 2013 with a full analysis of the issue and all 
the affected branches.” 

(emphasis added) 


It is somewhat disingenuous, in my judgment, for Mr Jenkins in this 
note effectively to blame others (which appear to include SPMs and 
the Post Office), rather than Fujitsu, for this problem not becoming 
known about until some years later. The trial did not explore 
communications between the Post Office and Fujitsu on this subject 
in any great detail, but SPMs are not left with a choice of whether 
to report something to the Post Office, or whether to report it to 
Fujitsu. One branch had a loss of approximately £9,800 (the exact 
sum was £9,799.88); some were £161 or less, and another had a 
gain of £3,100. 


Mr Coyne’s view is that this bug could have affected branches prior 
to Fujitsu’s investigations. He also suggests that it is unlikely that 
Post Office and/or Fujitsu have captured its full effects. My findings 
on that are as follows. Fujitsu appear to have conducted a thorough 
investigation in 2014, and there are spreadsheets showing the 
branches identified as having been affected. A code fix was 
produced in 2013, and this was first piloted to 50 branches on 21 
October 2013 and deployed to all branches on 29 October 2013. 
The bug could not have arisen prior to 2010 in any event as that 
was the date that Horizon Online was initiated. 


I do not know if any of the claimants in this litigation have losses 
which, when their claims are fully tried, they claim have arisen in 
the relevant period and as a result of this type of incident, but 
whose branches do not appear in the Fujitsu list. If so, that will 
have to be addressed at future trials. However, there is no evidence 
before the court currently of branches other than those included in 
the spreadsheet having been affected in the relevant period. 


I find that this is a bug with potential lasting impact, and indeed it 
did cause actual impact. 


4, Dalmellington bug/Branch Outreach Issue. 
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This is a Horizon Online bug. It was not expressly acknowledged as 
a bug in the Bug Table, but is effectively accepted by the Post 
Office as a bug, as it appears under the heading in its closing 
submissions as one of “the following bugs had transient impact.” It 
is therefore accepted as a bug by the Post Office in paragraph 6 of 
Appendix 2, but this is subject to the point regarding “transient 
impact”. For some time during the trial the Post Office would refer 
to this as “an issue”, and avoid use of the word “bug”. I find that it 
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is a bug - that much does not seem to be controversial, as 
paragraph 2 of the detailed part of Appendix 2 relating to this bug 
states the following: 


“This is a bug which Mr Coyne states has lasting impact on branch 
accounts. Post Office submits that there was no lasting impact on 
branch accounts.” 


It is therefore the nature of its impact on branch accounts that is 
now in issue. Its effects were experienced between 2010 and 2015, 
although it is named after the branch where its occurrence led to 
its discovery as a bug in 2015. Fujitsu performed an investigation 
as a result of the impact of this bug in 2015 at the Dalmellington 
branch. The investigation shows the following: 


Feb 2010 to Jan 2011 65 incidents 
2011 6 incidents 

2012 9 incidents 

2013 7 incidents 

2014 9 incidents 

2015 16 incidents 


The same document showed fixes applied in April 2010, January 
2011 and January 2016. It also showed only one call to SSC at 
Fujitsu in 2015, and none for any of the years 2010 to 2014 when 
the figures above show there were 93 occurrences of this bug. 


It is what is called a “cash remming error” and happened in certain 
circumstances. It was explained both by Mr Coyne, and further by 
Mr Godeseth in his evidence at [428] onwards, identified in the 
judgment itself. Basically, places where a full time branch Post 
Office is not justified, such as remote areas, may have such services 
on (say) half a day per week, in a mobile van or a village hall. These 
are called “outreach” branches, and they are connected to (or part 
of) a core branch, run by the same SPM. This means that the SPM 
is in charge of both the core branch and the outreach branch. The 
Dalmellington bug affected such branches. 


A PEAK was created on 13 October 2015 in respect of 
Dalmellington, namely PEAK PC0246949. The SPM in that branch 
had attempted to transfer cash to its outreach branches. The SPM 
sought to transfer £8,000 and ended up with a £32,000 transaction. 
This problem involved issues relating to two different scripts, the 
Post Log On script and the Forced Log Out script. The latter did 
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not close down the former correctly (or in software terms, did not 
lead to the former script correctly closing down the operation) 
which meant that the script was left on the stack of incomplete 
processes. If the SPM then logged back into the counter to perform 
a transfer of cash from their core branch to the outreach branch, 
the “pouch delivery” screen would still show, with “enter” being 
enabled. If the SPM were then to press “enter”, then the value of 
the transaction would repeat. As expressed in a confidential 
briefing given to the Post Office by Fujitsu: 


“It relies on a mechanism which checks the stack of incomplete 
processes to see if it is complete. Due to the fact that the stack is 
not empty (following the first problem) it thinks it has not finished 
and as a result attempts to repeat the last part of the script, which 
in this case is to record the remittance transactions and print the 
receipts” 


This would result in the amount of cash (in the case of 
Dalmellington, £32,000 as enter was pressed a further three times) 
being ‘Remmed Out’ of the core branch. Because each “pouch” 
which holds cash has its own ID, Fujitsu researched the number of 
times that duplicate pouch IDs had occurred in the BLE files. 


There are two points upon which the Post Office relies in respect of 
this bug. The first point is that the Fujitsu investigation in 
December 2015 showed that 112 potential occurrences of the 
Dalmellington issue had been identified in the previous 5 years. Of 
these 112 potential occurrences, 108 items were corrected at the 
time, either by Post Office issuing a Transaction Correction or the 
SPM reversing the duplicate Rem In. This left 4 remaining potential 
occurrences, for which Fujitsu had not yet established the outcome. 
However, the claimants rely upon two points of their own in this 
respect. One is that this showed that even though the bug had 
occurred multiple times and affected branch accounts, neither 
Fujitsu or the Post Office even knew it was a bug prior to late 2015. 
In my judgment, that is a valid point for the claimants to make. The 
second feature is that TCs are, as has been identified in the 
Horizon Issues judgment itself, outside of the Horizon System. 


A third feature is that the Post Office did not disclose the existence 
of this bug more widely. 


The Post Office also submits that “once the full extent of the 
Dalmellington issue was understood, Fujitsu produced a code fix 
without delay and it was rolled out in January 2016.” However, it 
cannot seriously be suggested that correction of the bug was 
“without delay” when the full history of the PEAKs are considered, 
with incidents going back to 2010. By the time this bug was 
remedied in early 2016, the mediation scheme had been brought to 
an end some time before (this occurred in March 2015) but some of 
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the branches affected by the Dalmellington bug were within the 
scheme, according to the Fujitsu investigation. However, none of 
the periods when the bug hit matched what were recorded in the 
investigation as “Mediation dates”. Not only that, but for the period 
up to the commencement of the mediation scheme, the Post Office 
was meeting a case brought by the claimants that there were 
widespread bugs within Horizon which impacted the branch 
accounts. Dalmellington shows that there was such a bug, which 
lay undiscovered (although its effects were apparent) until 2015. 


The Post Office also submitted the following: 


“As put to Mr Coyne by Mr de Garr Robinson, from an outsider’s 
perspective, you would not be able to tell the difference between 
someone remming in twice by accident, due to human error and 
someone remming in twice because of an error on screen.” 


I am not sure why this submission seeks, as it does, to deploy that 
submission as a point in the Post Office’s favour on the Horizon 
Issues. It is rather a point in the claimants’ favour, as it means that 
there was no visible sign that the bug had done what it had done. It 
could be misinterpreted as human error; its effects looked exactly 
the same, and Mr Godeseth in his evidence accepted this. It is also 
not a point in the Post Office’s favour that any known instances of 
this bug were corrected by means of TCs, or “by the SPM himself 
reversing the REM in some way”, which is how it was put in cross- 
examination. 


Two of the four unknown outcomes were for £25,000, and £2,500. 
The Post Office, in its closing submissions, explained why neither of 
these were in fact incidents of the Dalmellington bug because they 
were not outreach services, and also although bar codes for cash 
pouches were supposed to be unique, there was a small batch of 
them that were not. The documents from which these submissions 
were drawn show that the two branches with these discrepancies 
were those with FAD codes 209311 and 157242. However, the 
documents also report that both these branches had duplicate IDs 
for remming in of cash, that there had been “no contact from the 
branches raising issues for the period in question (February and 
March 2013)” and come to the following conclusion (although I 
consider it is worth noting that the SPMs in each of these branches 
do not appear to have been asked, nor does the ARQ data appear to 
have been used): 


“Post Office concludes the issues at the branches have arisen as a 
result of remittances pouches received at the branch entered 
manually which had the same barcode id. Thus creating duplicate 
entries which Fujitsu highlighted as part of the BLE files checks. 
However, in these instances from the available evidence Post Office 
concludes that the correct amount of pouches were delivered, 
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accepted and entered on Horizon. This is supported by the fact that 
there has been no negative impact in the branch accounts and no 
record of an issue raised by the branches with Post Office.” 


And “Moreover, these two branches have resolved in branch the 
issue encountered without referral to Post Office via NBSC and 
FSC or Fujitsu.” 


I find that this bug was present in Horizon Online from the date 
that Online was brought in, namely the year 2010. In my judgment, 
this is absolutely clear, and does not appear to be in dispute given 
the dates in the Fujitsu investigation. 


This bug would, in my judgment, have had an impact upon branch 
accounts - it is the extent of that impact that is in issue, whether it 
was lasting or transient. Dr Worden relied upon what he described 
as “a well-tested process of reconciliation and TCs to detect and 
correct errors in cash remming (used 20,000 times per year)” and 
said it was “straightforward for Horizon to detect any discrepancy 
between a “rem out” and the corresponding “rem in” (a mismatch 
arising either from a miscount, or a multiple count of a pouch) and 
then a TC can be issued.” He also added that “this process catches 
and corrects remming errors, whatever their cause - including if 
they arise from, or are provoked by, software faults.” He therefore 
implicitly accepted that the Dalmellington bug was a software fault, 
although he did not say so in terms. I find that it was plainly was. I 
also reject his evidence that it was “straightforward for Horizon to 
detect any discrepancy”; the evidence about this bug shows that 
Horizon did not detect it in these many cases. 


Dr Worden was also reliant upon the process of TCs to correct it. 
Further, the bug was there for 5 years and was not discovered, 
although its effects doubtless were experienced during that period. 


It is correct to record for completeness that all known impacts of 
this bug appear, on the face of the evidence before the court at the 
Horizon Issues trial, to have been corrected by the Post Office, with 
the exception of the two branches identified which were not 
outreach branches. The documents show that branch 209311 had 
no rem issue on, or about, the date of the duplicate ID incident of 
£2,500 (which was 1 March 2013) and the SPM had not raised any 
issue with the Relationship Manager Mr Andy Winn. There was no 
record of any TC having been issued for £2,500 and an audit in 
August 2013 had shown a shortfall at that branch of £350.55. 


The other branch, 157242, was operated by a SPM who transferred 
to a “Main agreement” in May 2014. Such an agreement did not 
feature in the Common Issues trial but I assume it means it is now 
a Main Post Office and not a branch. Cash rem in issues were 
shown at the branch in February 2013 but none of the same type as 
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the £25,000 duplicate rem in, and none on the exact date in 
February which was 18 February 2013. Although two of these were 
for very substantial amounts, none were for £25,000 and Mr Winn 
again said that no issue had been raised. I recite the findings in 
respect of both these branches for completeness. 


I find that this was a software bug that impacted upon branch 
accounts. In my judgment, it had lasting impact upon branch 
accounts. The fact that the occurrences of it (over 100 in a 5 year 
period) were corrected by means of TCs does not mean that it 
ought not properly to be characterised as a bug causing lasting 
impact to branch accounts. However, there were no accounting 
shortfalls ultimately experienced by SPMs whose branches suffered 
from the effects of the Dalmellington bug as these were corrected 
by means of TCs. SPMs whose branches suffered the effect of this 
bug were not therefore required to pay the sums the subject of the 
shortfalls to the Post Office on the evidence before the court. 


5. Remming In bug. 
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This is a Horizon Online bug. It was present for about five months 
in 2010 between March and August. It was not expressly 
acknowledged as a bug in the Bug Table. However, Dr Worden 
accepted it was a bug or defect in his cross examination on Day 20 
and it is accepted as a bug by the Post Office in paragraph 6 of 
Appendix 2, but is again said to be one that had only transient 
impact. The years it was said to have been present in terms of 
effect was “March - August 2010 and recorded as fixed approx. 
2011”. Mr Coyne said there were 14 branches affected. Dr 
Worden’s entry in the Joint Statement relied upon TCs, and stated 
“As for the Dalmellington bug, above - PO had a robust process for 
detecting and correcting remming errors, whatever their origin. So, 
there were no lasting effects on branch accounts.” In paragraph 2 
of the detailed part of Appendix 2 relating to this bug, the Post 
Office submitted that “any discrepancy would be transient as 
instances of this bug are caught by automatic reporting.” 

(emphasis added) 


The Post Office relies upon automatic reporting to catch incidents 
of this bug, and also maintains that Mr Coyne has conflated two 
issues, one relating to PEAK PC0203085 (which it terms “Issue 1”) 
and the other related to PEAK PC0195380 and other PEAKs 
associated with KEL acha4221Q (which it refers to “Issue 2”). Both 
of these relate to the remming in of cash, and both result in a cash 
pouch being recorded twice in error. However, they are said to be 
different issues because the sequence of steps taken by SPMs to 
trigger them are significantly different. 


The process of “remming in” works as follows. SPMs receive 
pouches of cash which are sent to the branch from the Post Office 
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cash centre. A certain amount of cash is held in each branch for the 
purposes of transacting Post Office business. Each pouch has a 
unique barcode that needs to be scanned by the branch when it is 
received. This automatically looks up the contents of the pouch so 
that the branch can confirm that the physical contents of the pouch 
match up to the record on Horizon. It is for this reason that each 
code is supposed to be unique. 


The way the system is supposed to work in theory is best taken 
from the Post Office’s closing submissions: “If there is a difference 
between what the system says is in a pouch of cash and the amount 
of cash actually in the pouch, the Subpostmaster should raise the 
issue with NBSC in order to get a Transaction Correction. A 
remming error leads to a mismatch between the amounts of cash 
remmed out to one place and the amounts remmed in from another. 
Remming errors are a violation of Data Entry Accounting and are 
picked up by Horizon.” 


That latter sentence was said by the claimants, after the written 
submissions were received, not to be present in any of the evidence 
used in the Horizon Issues trial. This point is dealt with in general 
terms at [120] above. In response to that challenge, the Post Office 
clarified its submissions and relied upon Dr Worden’s report, which 
was in general terms in respect of the general principle of double 
entry accounting; the HNG-X architecture document, which again 
demonstrated the theory or principle of the system; a PEAK 
document; and Mr Coyne’s cross-examination. It is necessary to 
consider those therefore, in order to see if the submission is 
substantiated. 


The principles of what is supposed to happen within Horizon, and 
the architecture, is not in dispute. Mr Coyne’s cross-examination 
passage that is relied upon does not support the general 
submission made to the PEAKs the subject of this bug, as it was as 
follows. I will quote the passage relied upon and the passages 
either side for context: 


“Q. So in forming a judgment on robustness you have first of all to 
see -- let's take a bug. A bug happens and the first question you 
ask is: did that bug or could that bug have had an impact on branch 
accounts? 


A. Yes. 


Q. Then you form a judgment on whether and to what extent the 
countermeasures in place in the Horizon system would have 
enabled that impact to be identified and fixed, yes? 


A. No, I think you would look at whether it did -- whether any 
countermeasure or control did prevent it from having a 
consequential impact, not whether it should have. 


Q. Well, whether it would have? 
A. Or whether it would have. 


Q. You don't consider whether it would have, you consider whether 
it did? 


A. Well, if it did there would be evidence that it did. It would be 
documented. 


Q. But in some cases it will be blindingly obvious, won't it, Mr 
Coyne? Let me give you an example, remming. A Post Office 
example. Remming in and remming out. Money is sent from 
Chesterfield to a branch. Chesterfield has a record of the money it 
sends over. The branch receives the money and types in -- or 
usually it is a barcode actually, but it records on the Horizon 
system how much money the branch has received. You are aware, 
aren't you, that there is an automatic system that checks 
Chesterfield's figures against the branch's figures? 


A. Yes. 


Q. Are you suggesting that every time over the last 20 years there 
has been a discrepancy between Chesterfield's figures and the 
branch's figures, are you suggesting that it is necessary for you to 
see the evidence of what happened next, of whether a transaction 
correction was sent and what the evidence was in relation to that 
branch? Are you really suggesting that's necessary? 


A. No, because in a typical scenario where the systems operate as 
they should, as they are designed, then the positioning of the 
countermeasures that you have put in place would pick up on that 
so that would work absolutely fine. In the scenario where the 
system doesn't operate as expected there is a bug, error or defect 
or communication fault, then a different set of scenarios will likely 
be encountered and it is understanding then what happens that is 
important. 


Q. Let me get this straight. You are suggesting that there are 
cases when you can take it as read, the situation is such that you 
can take it as read that a problem will be identified and it will be 
fixed. But you are suggesting there are other situations where you 
can't take it as read where specific evidence is needed, is that 
right? 


A. Yes. 


187. 


188. 


Q. Okay. But nonetheless, although you say that is necessary, it is 
your considered opinion that the Horizon system that you now see 
is robust, yes? 


A. Relatively robust, yes.” 


This evidence does not support the submission that remming errors 
are picked up by Horizon. It is necessary therefore to look at the 
actual PEAKs, to see what they show. The one associated with what 
the Post Office called Issue 1, PC0203085, is dated 22 August 2010 
and is headed “pouch remmed in on two counters at same time”. 
The first entry under impact statement is: 


“The same pouch can be remmed in to the system more than once, 
resulting in a shortage at the branch which POL have to rectify by 
issuing a Transaction Correction. 


1. call has to be processed, and corrective action taken, by SSC, 
MSU and POL 


2. visible to POL and the branch when it happens 
3. very rare” 


In my judgment, that entry alone is evidence of a bug. It shows a 
pouch can be remmed in more than once - admittedly rarely - and 
that a TC is necessary to correct this. However, in order to be 
clear, the PEAK goes on for some pages - 13 in total - and includes 
contradictory entries within it from Fujitsu such as the following 
two. One is on 17 August 2010 from Anne Chambers: 


“A cash pouch was remmed in twice at branch 126109: 

Pouch barcode 399347067204 

2p coin £60 

50p coin £250 

5p coin £100 

Session 1-350379 16/09/2010 10:08 

Session 2-195226 16/09/2010 10:08 

The PM cannot reverse the transaction since rem reversal isn't 
allowed. 

This is NOT another example of the duplicate rem problem that we 
have seen in the past, where use of the Prev key accepted the same 
pouch twice. In this case the pouch was processed on both 
counters... 

09:05 c2 get pouch status, retrieve pouch details 

09:06 c1 get pouch status, retrieve pouch details 

09:08 c2 settle pouch delivery 

09:08 c1 settle pouch delivery 

There were some printer problems on counter 2 which probably 
explain why this was done.” 
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The other contradictory entry within the same PEAK is the next 
day, 18 August 2010 and states, again from Anne Chambers 


“I checked whether there were any exceptions in the BAL OSR logs 
for any of the messages, there was nothing. 


Gareth Jenkins thinks that it should not be possible to complete the 
rem in on both counters. Please investigate.” 


Under Root cause and solution, the following entry is made on 19 
August 2010: 


“When an auto remin pouch id is settled successfully, the system 
updates the 

COUNTER READ TIMESTAMP in LFS RDC HEADER to a not null 
value for that pouch id. 

The race condition for auto remin pouch delivery is handled at 
SettlePouchDeliveryServiceSettlementProcessor.processPouch(). 
This method checks’ during settlement whether the 
COUNTER READ TIMESTAMP in LFS RDC HEADER is null or not 
null value. 

If null, the pouch id is good and settlement completes successfully. 
If not null, the pouch id is already processed and error is thrown. 
The query that gets the COUNTER READ TIMESTAMP from 
LFS RDC_HEADER is 'SettlePouchDeliveryPreCheck'. 

In this query, the input parameter for pouch id is defined 
incorrectly. 

It is given "pouchBarcode[String]", but in dyno the pouch id is 
"pouchId". 

This is the root cause why the query always returns null although 
the 

COUNTER READ TIMESTAMP is not null. 

The solution is to define the pouch id input param to 
"pouchId[String]" in 

'‘SettlePouchDeliveryPreCheck'. 


A fix is then identified, stated to be a low risk and low complex 
issue, which also states: 


“LIST OF LIKELY DELIVERABLES: 
Updated sql of SettlePouchDeliveryPreCheck”. 


In my judgment, this PEAK is evidence of a bug and a fix is 
required to remedy it. It also shows that remming in errors are not 
always picked up by Horizon. 


The second issue is related to what are different PEAKs, namely 
PC0195380 and other PEAKs associated with KEL acha4221Q. I 
will not deal with them all, but these relate to what the Post Office 
terms Issue 2. This KEL was raised by Anne Chambers on 2 March 
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2010, updated on 3 May 2011, and the keywords are remittance, 
remittence, remin, pouch and delivery. It can be seen that the 
following is included in the KEL: 


“Symptoms 


The clerk went into the Delivery menu and scanned two pouches 
(one of currency and one of coins). The second pouch was recorded 
twice on the system, resulting in a loss of £80. 


Two Rem In slips relating to the second pouch were output, both 
identical, as well as one for the first barcode. 


In the most recent instance, the same pouch was remmed in on two 
different counters at about the same time. 


Problem 


This was caused by using the Prev key during / just after the pouch 
barcode scans. Now fixed - details in PC0195380. 


In PC0203085, the same pouch was processed on both counters... 
09:05 c2 get pouch status, retrieve pouch details 

09:06 c1 get pouch status, retrieve pouch details 

09:08 c2 settle pouch delivery 

09:08 c1 settle pouch delivery 


PC0203085 - fix applied Jan 2011. 
Solution - ATOS 


A transaction log search will prove that the Rem In has been 
duplicated. 


Send call to SSC, quoting this KEL. 
SSC: 


Known problems have been fixed, so needs fresh investigation if it 
happens again. 


POL may need to issue a TC to undo the effects of the extra rem in 
(in the meantime the branch will report a shortage), so MSU need 
to inform POL via BIMS.” 


Given that the PEAK on Issue 1 is dated August 2010, and the KEL 
itself was raised in March 2010, the KEL must have been updated 
with the entry relating to that PEAK and that the fix was applied in 
2011. Those entries cannot have been included in the KEL 
originally, as they post date its creation in March 2010. This means 
therefore that someone at Fujitsu obviously concluded that the 
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later PEAK in time (on Issue 1) was relevant to the same KEL (on 
Issue 2) as it was added to the KEL. I agree with whoever at Fujitsu 
linked these PEAKSs to the same KEL. I consider that the KEL must 
refer to broadly the same issue, namely the remming in of pouches 
that were counted twice by the Horizon System. A fix was in any 
event issued for the issue identified in the KEL. However, it is 
correct and I accept the Post Office’s submissions that upon closer 
analysis it can be seen that there are two separate, but very similar 
in effect and related, issues here, both of which have the same end 
result, namely the duplication of remming in of cash pouches. 


Finally, in the Post Office’s clarification or response to the 
challenge brought by the claimants to the submissions on this bug, 
(which complained that there was no evidence to support the 
submission that remming in issues would be automatically “picked 
up” by Horizon), the Post Office referenced another document at 
{F/624/1}. It stated that the document was PC0197838, but in fact 
the reference is BE/0197828, as it is a BIMS Incident Report and 
not a PEAK, and is dated 16 April 2010. The reference number is 
the same but the initials are BE and not PC because it is a different 
type of document. However and in any event, the document relied 
upon by the Post Office identifies an exception type as “Receipts 
and Payments do not balance (post migration)” but it does deal 
with the same issue as this one, as it states within the body of the 
document that “The PM has used the Previous key whilst scanning 
in pouches, which has caused a duplicated rem” and also: 


“The two identical transactions each comprise of: 

Prod ID = 6287 (Rem In Cash from AD); Qty = 1; Amount = £1680 
The Post office Counter log shows this being made up of: 

£100 of 20p coins 

£80 of 2p coins 

£500 of 50p coins 

£100 of 5p coins 

£500 of £1 coins 

Action for Post Office Ltd: 

Post Office Ltd will need to contact the PM to confirm if the PM has 
two identical Rem In slips for Session 489165, both showing Pouch 
ID 399345656721. 

If this is the case, then the Rem In has been duplicated and this 
duplication needs to be reconciled, possibly by using a Transaction 
Correction.” 

(emphasis added) 


I accept that the issues on this bug can be split into Issue 1 and 
Issue 2 in the way that the Post Office suggest. If I am wrong about 
that, then the PEAKs I have identified above, including the BIMS 
Incident Report, are evidence of and relate to the same bug. Given 
the finding of two issues, then Issue 1 and Issue 2 could either 
relate to different iterations of one bug, or to two different bugs. 
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These bugs or this bug (depending upon which semantic analysis of 
Issue 1/Issue 2 is correct) both impact branch accounts. Issue 2 
related to the pilot phase of Horizon Online. However, neither of 
these issues are caught by automatic reporting. The Post Office 
accepts that Issue 1 was reported to it by the SPM affected. The 
fact that the BIMS report requires the Post Office to “contact the 
PM to confirm if the PM has two identical Rem in slips” for the 
session identified makes it clear that this also applies to Issue 2 as 
well. There is therefore, in my judgment, no practical difference 
whether one approaches this as two different issues with the same 
effect, or one. 


On either analysis, it is a software bug and I find that these or this 
impacted upon branch accounts. In my judgment, it had lasting 
impact upon branch accounts. The fact that the occurrences of it 
were corrected by means of TCs does not mean that it ought not 
properly to be characterised as a bug causing lasting impact. 


6. Remming Out bug. 
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This arises under Legacy Horizon. The Post Office’s submissions 
effectively accept that there was a bug under this heading, because 
in the detailed entry in its Appendix 2 it recites the following: 


“Mr Coyne states that Bug 6: Remming Out is a bug with lasting 
financial impact. It comprises two separate issues, only one of 
which was a bug. Any discrepancy caused by either issue would be 
transient as instances of both issues were caught by automatic 
reporting.” 

(emphasis added) 


It was split into two in the Bug Table, 6(i) which was identified as 
KEL acha508S and 6(i) which was identified as KEL 
GMaxwell3853P. The former was identified as February/April 2007, 
recorded as fixed approx. 2007, with Mr Coyne identifying 57 
branches affected; and the latter was in May 2005 with one branch 
being affected. They were both remming out issues, hence the 
grouping by the experts in the bug table. It was not acknowledged 
as a bug in the bug table. However, it is now accepted as a bug by 
the Post Office in paragraph 6 of Appendix 2, but is said to have 
had transient impact. 


Again, for both of these iterations, 6(i) and 6(ii), Dr Worden used 
the same wording as he had in the entry for bug 5, the Remming In 
bug, namely “As for the Dalmellington bug, above - PO had a 
robust process for detecting and correcting remming errors, 
whatever their origin. So, there were no lasting effects on branch 
accounts” (emphasis added). It was also said by the Post Office in its 
submissions that Mr Coyne had conflated two unrelated issues under 
the heading “Remming Out”. Given this was an entry in the 2" Joint 
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Statement agreed by the experts, it is a little unfair to state that Mr 
Coyne had done this, as the entry was obviously agreed by Dr Worden. 
However, it was split into 6(i) and 6(ii) and I will consider each 
separately. 


Issue 6(i) arises as follows. What is called “a remming error” leads 
to a mismatch between the amounts of cash remmed out to one 
place and the amounts remmed in from another. The Post Office has 
submitted that remming errors are a clear violation of Data Entry 
Accounting and are picked up by Horizon. The two different issues 
are as follows. 


As the obverse of the coin of remming in, SPMs rem out pouches of 
cash to be returned to the Post Office Cash Centre. A single pouch 
may contain multiple bags of coins or cash and each bag can only 
hold one denomination, and there is a limit on how much cash can 
be placed into a pouch. The cash can be remmed out before it is 
physically collected. When remmed out the cash appears in a 
different line in the branch accounts. On collection, the collection 
team scan a barcode on the pouch and the cash is removed from 
the “cash in pouches” line of the accounts. 


When remming cash out, branches should have made one entry for 
each denomination and value and, if there were multiple bags for a 
particular denomination, the quantity of bags should have been 
specified in that single entry (e.g. 2 x £500 of £2 coins). However, 
if the SPMs had made multiple entries for each denomination and 
value (e.g. one entry for 1 x £500 of £2 coins and then a second 
entry for 1 x £500 of £2 coins), Horizon would only record the first 
bag as having left the branch’s cash holdings, but all of the bags 
would show on the “cash in pouches” line. This would create a 
discrepancy in the branch accounts, because all of the cash would 
have been collected. 


This issue occurred as a result of Release T30 INC1 (which is 
referred to in the Fujitsu Service Review Book of February 2007 
covering that month) and the Post Office’s submissions accept this. 
A further entry for Monday 12 February 2007 in the same 
document states: 


“The issue was found to be when Post Masters would REM out 2 
identical items instead of using the quantity button. As this was an 
error in the code, the counter 36 6 s/w was regress overnight. 
After the regression of the s/w overnight, it was then identified that 
49 branches had an issue with the REM out process causing a cash 
imbalance at the counter.” 


I consider this to be a bug. Fujitsu created KEL acha508S to advise 
branches to correct the problem manually and ran automated 
reports to spot any further occurrences. Fujitsu then made 
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changes to Release T30 INC1 and rolled it back out to fix the issue. 
However, that the bug was fixed does not, in my judgment, mean it 
ought to be ignored. Paragraph 22 of the Appendix 2 submissions 
on this by the Post Office submits “in any event, it was picked up 
automatically”. This is not correct. An entry in the February Service 
Review Book on page 11 of 33 of that document shows that there 
were 526 calls to the Horizon Service Desk as a result of this 
specific issue, of what were said to be a high number of software 
calls in that month of 3792. That does not support a submission 
that it was picked up automatically. 


So far as Issue 6(ii) is concerned, the following evidence from Dr 
Worden - which is entirely ignored in the Post Office’s submissions 
in Appendix 2 - is relevant. 


“Q. So we are looking at 6(ii). 

A. Yes. So again my standard wording comes in. 

Q. That's it. You will see there: 

"Remming out' (ii) Bug (not acknowledged)." 

This isn't one that you had in either of your tables in your appendix 
originally. 

No. 

Q. Did you have a look at the KEL in that case? 

A. I believe I did, yes. 

Q. Can we look at it now, please. This is at {F/276/1}. 
A 

Q 


a 


. Rem out... Yes. 

This is the GMaxwell3853P KEL. If we look at the bottom 
where it says "Solution - Atos". 
A. Bug in the code, yes, right. 
Q. "Development have identified a possible bug in the counter 
code. However, due to the frequency of the problem & the risks 
involved in making the necessary changes it has been decided that 
a code change will not be made." 
A. Yes. 
Q. You hadn't noticed that, had you? 
A. I believe I had, but again the point about permanent effects, 
lasting effects, on branch accounts is not altered by that. 
Q. And the point about this is there was a lasting bug? 
A. There may have been a lasting bug, but they took a decision 
based on its frequency and the difficulty of making the change, 
perhaps the risk of making the change, that it was not worth 
addressing. That actually the remming process would pick it up 
and its frequency -- they took a decision, I don't know what the 
details were of their trade off. They took a decision that fixing the 
code was not worthwhile. 
Q. I'm just asking you: it is a lasting bug? 
A. Itis a lasting bug, yes. 
Q. Let's go to number 7 now, if we may, please.” 
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There was then an interjection by counsel for the Post Office who 
drew attention to possible misunderstanding in terminology given 
the use of the phrase “lasting bug”. Dr Worden then added the 
following, when asked by the claimants’ counsel whether he had 
understood what was meant by the question: 


“A. I did. And I'll clarify what I mean: it's a bug that was not fixed, 
we cannot see was fixed, but it was a bug without lasting effects in 
my opinion.” 


In my judgment, it was clearly accepted by Dr Worden as a bug, 
and the entry in the KEL itself also says there was “a possible bug 
in the counter code” but a decision was taken not to change the 
code because there was only one instance of this occurring. 


I find that 6(ii) also relates to a bug and reject the Post Office’s 
submission that “it wasn’t a bug”. 


The claimants’ case on the Remming Out bug in the Bug Table at 
item 6 is therefore made out. Both of these - issue 6(i) and 6(ii) - 
are, in my judgment, software bugs. I find that these had potential 
lasting impact upon branch accounts. 


7. Local Suspense Account issue, not the same as 3. Suspense Account 


bug. 
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This was reported in 2010 and recorded as fixed in September 
2010. It arose in the early days of Horizon Online, during the pilot 
scheme. It was not acknowledged in the Bug Table, but is now 
accepted as a bug by the Post Office in paragraph 6 of Appendix 2, 
although it is one of those bugs said to have had transient impact. 
Mr Parker had identified 33 branches affected. Mr Coyne recorded 
what he said were four associated KELs, namely acha5259Q (for 
which there were 6 PEAKSs) (this KEL is mistakenly recorded twice 
in column 3 of the Bug Table, with acha5838T only mentioned in 
the text in that column); cardc2043L (10 PEAKs); PorterS199P (3 
PEAKs); and acha5838T (which states there are “two different but 
similar problems” and appears in the text, but not in the list of 
KELs at the end of the text by Mr Coyne). Dr Worden’s comments 
in the table were that: 


“The KEL acha5259Q implies that PO and Fujitsu were able to 
identify all occurrences of the problem, without being notified by 
any Subpostmaster. I would therefore expect them to have 
corrected any impact on branch accounts as part of normal error 
correction processes. 


I would not expect evidence of all corrections to accounts to have 
survived to the present day. PEAKS and KELs are not used to 
record corrections of financial impact.” 
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He also relied upon a statement by Mr Parker that there was 
“Temporary financial impact which would have been cancelled out 
in the following period by a corresponding discrepancy”. To use the 
Post Office’s own submissions it was “an intermittent system issue 
which temporarily prevented branches from rolling over into the 
next Trading Period”. 


The statement of Mr Parker introduces another concept, similar if 
not identical to Dr Worden’s “transient impact”, and that is 
“temporary financial impact”. Dr Worden is correct that PEAKs and 
KELs do not record corrections of financial impact; they do 
however record financial impact, because when an SPM reaches 
SSC (which raises the PEAK) they often start by recording “SPM 
says he/she has a problem in that...... ” and financial impact is often 
then recorded. The Post Office in paragraph 2 of the detailed part 
of Appendix 2 dealing with this bug states that “any discrepancy 
would not be lasting”. This does therefore accept, albeit implicitly, 
that it would have an impact upon branch accounts. It is also 
submitted that it was a “teething problem” from the early days of 
the Horizon Online pilot scheme. 


The cross-examination of Dr Worden on this bug was as follows, 
and I find to be highly relevant. 


“Q. Then you say: 

“T would not expect evidence of all corrections to accounts to have 
survived to the present day. PEAKs and KELs are not used to 
record corrections of financial impact." 

A. Yes. 

Q. Fujitsu, in fairness, analysed the KEL, you say here, and said: 
".. 'Temporary financial impact which would have been cancelled 
out in the following period by a corresponding discrepancy." 

Do you see that? 

A. Yes. 

Q. Now, did you look at the KEL? 

A. Well, I infer from that Fujitsu did it, but I had done it previously. 
Q. Let's go, please, to {F/637/1} which is the KEL acha5259Q. 

A. Right. Sorry, can I remind myself of the ..."... local suspense 
cleared ... balance report ... cash ..." 

Q. Do you see if we look at the problem -- 

A. Yes. 

Q. -- the summary was that the PM was forced to clear local 
suspense several times resulting in the cash figure on the balance 
report changing and possibly a discrepancy in the new trading 
period. 

A. Yes. 

Q. So that certainly had the potential to cause an impact on branch 
accounts at the time, subject to later correction? 

A. Yes. 

Q. And probably did, yes? 
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A. I think it did, yes.” 


It is in my judgment self-evident, that if the branch accounts had to 
be corrected later, in a new trading period, that the accounts for 
the period when the effects of the bug were experienced were 
wrong. This was a bug. It was a bug that had a lasting impact upon 
branch accounts as I have defined that phrase. The fact that the 
discrepancy was later corrected, in the words of Mr Parker a 
discrepancy “which would have been cancelled out in the following 
period by a corresponding discrepancy” makes this entirely clear. 


8. Recovery Issues. 
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These are Horizon Online issues. The Post Office does not accept 
these are bugs at all. These are not agreed as bugs by the experts, 
and there are four different types included in the table, with years 
of effect from 2010 to 2018. Mr Coyne’s entry stated that “The text 
within the PEAKS and KELs suggests that in each case a branch 
account discrepancy would be evident and would require 
correction by the Post Office.” Dr Worden stated that: 


“The KELs and PEAKs cited by Mr Coyne are not indicative of 
errors in Horizon. They provide guidance on how to correct 
discrepancies caused by human errors or other errors in 
transaction recovery (‘recoverable transactions’) 


Because there were many such errors, there were many calls to the 
help desk and many PEAKs and KELs. Normally, correction of 
errors involved back office reconciliation and issuing TCs. This was 
accurate and effective; I have derived an upper limit of £2 per 
branch per month on the mean impact of erroneous TCs. 


One important KEL acha959T was guidance to the back office 
MSU, not for Subpostmasters”. 


There was a minor typographic error in the submissions in 
paragraph 4 of the detailed part of Appendix 2 which referred to 
bug 9. However, the four different types of recovery issues are 
addressed in the subsequent paragraphs and the conclusion 
paragraphs deal with bug 8. This was later corrected in a sheet of 
corrections, which clarified that paragraph 4 of the detailed part of 
Appendix 2 should have stated "Mr Coyne states that Bug 8: 
Recovery Issues is a bug with lasting financial impact. Post Office 
submits that it is not a bug at all". 


Recovery deals with the situation where there has been an 
interruption, which means when a basket of transactions is 
interrupted during the course of dealing with a customer. This 
could be for several reasons including a power failure, system 
crash or communication failure between the branch and data 
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centre. This is a risk which has to be dealt with as it cannot 
sensibly be eliminated in a system such as Horizon. Horizon runs 
various automated reports each day to look for failed recovery 
events. Mr Coyne dealt with two different documents in his report, 
PC0197769 and KEL acha959T. The former was created on 15 April 
2010 during the pilot phase of Horizon Online. The root cause of 
the issue was identified on 26 April 2010, work on a fix was started 
and this was released in early June as shown in Release PEAK 
PC0199000 (and then estate-wide to pilot branches) by mid-June 
2010. It did however undoubtedly evidence a bug. The reference to 
the “959T” KEL below is the KEL whose full title is acha959T which 
was raised by Anne Chambers. 


KELacha959T is also the very same KEL referred to in PEAK 
PC0214226 dated 14 December 2011, headed “Failed Recovery 
Transaction(s) 12 Dec 11” which I have considered at [110] and 
[111] of the substantive judgment, as it related to Mr Tank’s 
specific experience at his branch regarding the £195 transaction. I 
have already dealt with one entry in that PEAK in that part of the 
judgment. I have accepted Mr Tank’s evidence on this incident. 
That PEAK was closed with the categorisation “Final - Advice after 
Investigation” and “Root Cause - General in Procedure”. The entry 
at 10:40:51 on 14 December 2011, also from Mr Bragg, shows the 
transaction details which plainly identify an authorised transaction, 
a failure of the basket settlement, the failure of repeated retries by 
the system and the failed recovery. It was this analysis that 
preceded Mr Bragg’s conclusion that “the customer’s account 
should be correct but the branch will have a shortage (for a 
withdrawal) because the session hasn’t been recorded”. This is 
plainly a fault in Horizon, and is directly contrary both to the Post 
Office’s case on failed recoveries in the Bug Table, and also its case 
on Mr Tank’s evidence. 


Dr Worden’s cross-examination on the issue of recoveries, by 
reference to entry 8 in the Bug Table, was also highly relevant: 


“Q. Let's move if we may now to number 8 and we find that -- just 
to do it the same way again, if we go back to the joint statement 2 
table and we look at {D1/2/7}. 

Row 8. 

. Itis row 8. The recovery issues are identified there. 

Mm. 

. Mr Coyne has made references to the text in certain PEAKs. 

A. Yes. 

Q. And you have given your opinion there, you say they are not 
indicative of errors in Horizon, they provide guidance on how to 
correct discrepancies caused by human errors or other errors. 

A. Yes. 

Q. You say: "Because there were many such errors, there were 
many calls to the help desk and many PEAKs and KELs." Then it is 
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basically your same theory, that: “Normally, correction of errors 
involved back office reconciliation and issuing TCs." 

And that's accurate. 

A. It is a different theory really. Recoverable transactions are a 
big subject and they are complicated because the point at which a 
recoverable transaction can go wrong is very variable through the 
sequence, and therefore the number of recovery actions, the type 
of recovery actions is complicated. Horizon was designed so that 
with the assistance of the postmaster most of these could be 
recovered, but there are things called failed recoveries, which 
again were part of the design of Horizon, and those were the failed 
recoveries but particularly the ones where Fujitsu had to get 


involved. But failed recoveries means the recovery process had 
failed, that's the way Horizon was designed because these things 


are so complicated you can't handle them all automatically. So it is 
a big subject but there is plenty of useful evidence about it. 

Q. You say in your table at {D3/2/124}. 

A. This is the 959T KEL, is it? 

Q. Yes. You say: 

"This is another complex KEL with strong overlap with previous 
KEL." 

A. Yes. This KEL is cited in a large number of PEAKs. 

Q. And in fact when we go back to the table at {D1/2/7}, Mr Coyne 
has actually quoted from one of the PEAKs we see there, the first 
one he quotes from. Do you see "PC0198352"? 

A. Yes. 

Q. And he has quoted: "This problem caused a loss at the branch 
for which they should not be liable." 

Pausing there. This did have the potential to cause the type of 
discrepancy which you have given evidence about being later 
corrected, didn't it? 

A. The word "problem" doesn't imply a bug in Horizon. 

Q. You didn't read these PEAKs yourself, did you? 

A. I have read a fair sample of recovery PEAKs. Maybe not these 
ones, but I have read a fair sample. 

Q. If we go over the page {D1/2/8}, do you see Mr Coyne points 
out there that this KEL is referred to by 2,473 PEAKs -- 

A. Yes, that's -- 

Q. -- from 2010 to 2018. 

MR JUSTICE FRASER: Where are we looking? 

MR GREEN: At the top of that page under the "Coyne Opinion as 
to branch account impact", my Lord. 

MR JUSTICE FRASER: Yes. 

A. That is correct. 

Q. Just pausing there, Dr Worden. If something is handled by 
NBSC and corrected straightaway, it doesn't make it necessarily up 
the line to SSC, does it? 

A. No. I mean there are various levels of recoverable transactions. 
There is a recoverable transaction that the subpostmaster can 
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follow the instructions and can fix it himself, there is the ones 
where he needs help from NBSC, there's ones where he needs help 
from further down, and there are failed recoveries where PEAKs 
are involved. All of those things happen. 

Q. In the case of a failed recovery, it can be sorted by NBSC in 
some way or by a transaction correction without referring to SSC? 
A. That's a good question. I think the failed recovery port usually 
involves not just NBSC, it involves MSU, and there is quite a 
complicated process of guiding the transaction through various 
states until it gets to the F99 state. 

Q. Can I just invite you to look at what Mr Coyne said -- 

A. Yes. 

Q. -- having actually tried to identify through the available 
disclosure how many PEAKs were referenced and set them out. He 
says it is referred to by 2,473 other PEAKs from over that range: 

" .. each of these may (but may not) indicate a similar issue with 
the Horizon recovery process and potentially creating a 
discrepancy within branch accounts." 

If we accept his definition of discrepancy and not yours, yes? 

A. This is all about temporary discrepancies in branch accounts. 

Q. Park the temporary discrepancy point about which we have 
heard a lot. 

A. Yes. 

Q. What he has said there is correct in the number of PEAKs as far 
as you know? 

A. It is correct, yes. 

Q. And it is scrupulously fair in how he has described it? 

A. Well, "issue with the Horizon recovery process", I believe these 
issues are how Horizon was designed. 

Q. But what he's said there, he has carefully put "issue", leaving 
open the availability of an argument between experts about what 
the nature of the issue is, but what he has scrupulously done there 
is perfectly fair, isn't it? 

A. I think so. I mean the difference between the experts is that, to 
summarise something Mr Coyne said in his oral evidence, he 
believes that these recovery processes can be automatic. Now, I 
don't believe that. I believe recovery of recoverable transactions is 
very complicated and needs human intervention and things like 
failed recoveries are inevitable.” 

(emphasis added) 


I prefer the evidence of Mr Coyne that recovery processes can be 
automatic in systems, and I also accept that “failed recoveries” as 
they were defined above are inevitable. The issue here is whether 
the Horizon System process for dealing with the recovery 
processes when they had in fact failed was working as it should, or 
defectively. The fact that there are so many PEAKs identified in the 
KEL, and the contents of those PEAKs clearly demonstrates, in my 
judgment clearly, that it was not. I reject Dr Worden’s evidence 
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that human intervention should be required or expected for some, 
most or all failed recoveries. I also reject his evidence about the 
word in the PEAK “problem”. He said “The word "problem" doesn't 
imply a bug in Horizon.” In my judgment, on this subject, it clearly 
does. 


The Post Office’s own submissions set out how trading periods 
were affected. I shall reproduce them: 


“10. The issue is described in KEL acha5650L. It involved 
recovered transactions being written to a different trading period 


than the original transaction. In summary: 


i) if a transaction failed in one trading period and the recovered 
transaction went into the next trading period, there would be 
a loss in the first trading period 5 and a corresponding gain in 
the next trading period such that there would be no overall 
loss in the branch; and 


ii) if the recovered transaction was written to an earlier trading 
period, there would be a loss in the current trading period 
but no corresponding gain because the previous trading 
period would have already been rolled over again. There 
would be a net loss in that scenario.” 


(emphasis added) 


That demonstrates that branch accounts would be affected. In the 
first possibility, the accounts would be wrong for one trading 
period (until the so-called “corresponding gain” occurred in the 
next period) and in the second, it is accepted there would be a net 
loss. 


In my judgment, the issuing of a later TC is neither here nor there 
in this scenario. Where there has been a failed recovery, this is now 
flagged automatically by Horizon to SSC at Fujitsu for 
investigation. So far as the modern or more recent way that the 
system deals with failed recoveries, it is plainly the case that 
Horizon now automatically flags these. The fact that it did not do so 
previously, in my judgment, and the existence of all these PEAKs on 
this subject, clearly demonstrate a bug, error or defect, and that 
conclusion is not avoided by the fact that a later TC was issued. 


On 16 January 2017, the automatic monitoring service spotted a 
failed recovery and PEAK PC0256502 was raised by Fujitsu. A 
BIMS was issued to Post Office to correct the discrepancy. On 17 
January 2017, the SPM in question contacted the helpline 
regarding the same issue. A second PEAK PC0256566 was raised. 
As quoted by Mr Coyne in his report, this PEAK clearly records that 
this issue was already identified by Fujitsu and resolved under the 
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earlier PEAK. Hence the later PEAK in time was closed. However, 
although this example required corrective action by the Post Office 
(not by Horizon) it is not evidence of a bug. The former PEAK 
namely PC0197769 did evidence a bug. I also find that PEAK 
PC0214226 is evidence of a bug. 


Mr Parker’s evidence also makes it clear that this bug had an 
impact upon branch accounts. Mr Tank’s evidence demonstrates 
the same, albeit for a different date and time. This impact would 
persist until it was later corrected. 


9. Reversals. 
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This is a Legacy Horizon bug and occurred in 2003. It was not 
acknowledged as a bug in the Bug Table but is now accepted as a 
bug by the Post Office in paragraph 6 of Appendix 2. It is however 
said only to have had transient impact. 


Dr Worden’s entry in the 2™ Joint Statement in relation to this 
stated “Transaction reversals are a complex area which, like 
recoverable transactions, are less familiar to Subpostmasters and 
are more prone to human error. They lead to many calls to the help 
line and to many KELs and PEAKs - not necessarily related to any 
fault in Horizon.” This can be seen as an example of Dr Worden 
attributing fault to the SPMs rather than accepting the presence of 
a bug, but given it is now accepted as a bug it is not necessary to 
consider that any further. Mr Coyne’s entry in the 2™ Joint 
Statement stated “In April 2003 due to a failure in regression 
testing, Horizon version S30 was released by Fujitsu and this 
introduced a bug where the value of transactions reversed by 
Subpostmasters was shown twice in the amount of the reversal in 
branch accounts.” 


This was clearly a bug. A Fujitsu document, PEAK PC0089918, 
identifies an issue in which attempted reversals of remming in 
transactions resulted in the magnitude of the transaction doubling, 
rather than the transaction being reversed. Initially, this is blamed 
upon the SPM in question and the Post Office in its Appendix 2 
continues this, submitting that “The Subpostmaster (branch 
FAD003227) was trading in Stock Unit Y and remmed in £13,910 of 
cash. He continued to trade in that stock unit in error, instead of 
moving to Stock Unit I. He therefore attempted to reverse all of 
the transactions he had made in Y (including the rem in). Instead 
of Y showing a balance of zero, the rem in had doubled to show a 
discrepancy of £27,820. Having spoken to NBSC,_ the 
Subpostmaster attempted further reversals but these failed and 
produced an error message indicating that the initial reversal had 
been completed successfully.” 
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However, it was clearly caused by a bug. This is effectively 
accepted later in the submissions which state, correctly in my 
judgment, that “the issue was caused by a software error, which 
had been introduced as part of implementing the fix for PC0083954 
in Legacy Horizon, and which incorrectly calculated the reversal 
value and quantity (essentially, the wrong mathematical sign was 
applied when reversing RIAD transactions (+ instead of -))”. 


If a minus arithmetical function is required, namely subtraction, 
but by software fault this actually occurs as an addition, then this is 
clearly a bug. It is certainly an error or defect in the system. This 
was then compounded by PC0083954, a fix which introduced a 
code change to ensure that a partial settlement of cash always tried 
to reduce the stack value. However, that fix did not work when 
reversing a rem in and so the solution was to pass the mode to the 
function also. 


A fix was required, and because of the potential impact on live 
branches, the development of a fix is said to have been “fast- 
tracked”. The Post Office submissions stated that “within 15 days of 
the issue first being raised, Fujitsu had implemented a fix to 2,178 
branches. It is not known precisely when the fix reached all 
branches in the estate but Fujitsu believe it is likely to have been 
before 2 June 2003....” That again comes perilously close to 
introducing evidence by the back door through submissions. 


This was clearly a bug with potential lasting impact to branch 
accounts. The fact that the impacts of its occurrence were 
corrected by means of TCs does not affect its existence. It occurred 
for a short period in 2003. There is no evidence before the court of 
branches being affected by this outside that period, or of branch 
accounts affected by the bug that were not later corrected. 


10. Data Tree Build Failure discrepancies. 
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This is a Legacy Horizon bug. Its identified effect was during 1999 
and 2000, so the very early days of Horizon. However, KELs 
relating to the PEAKSs in those dates were created in July 2005 and 
updated in November 2007. A data tree is an accounting node 
hierarchy, or a cash account node hierarchy. 


It is accepted by the Post Office in paragraph 7 of Appendix 2 as 
one of a number of “bugs with lasting impact (although they were 
resolved)”. The PEAK dealing with this reads “Data trees have been 
failing to build fully, and the system has not been detecting this.” 
Data trees were part of Legacy Horizon, and were used to build a 
summary (or picture, the word used by the Post Office in its 
Appendix 2) of the accounts. The building of data trees is a 
software function, and given “the system” is Horizon and is 
supposed to detect failures of this nature, it is difficult to see how it 
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could ever have been in issue that this was a bug. Mr Coyne’s entry 
in the Bug Table recites that “Dugannon branch suffered a £43,000 
discrepancy but the cause was not immediately known. £52,814.29 
at the Yate Sodbury Branch. £9,368.40 at the Appleby 
Westmoreland branch.” These are sizeable sums and arose in the 
branch accounts. That this is a bug was de facto accepted by Dr 
Worden in the Bug Table due to the text of his entry which states: 


“There was a bug which has potential impact on branch accounts, 
early in the lifetime of Horizon. Soon after it arose, the error was 
trapped and detected by DEP and was then soon fixed. 


The fault was easily noticeable at branches before the error 
trapping which was soon introduced and would be even more 
noticeable after that. Only three branches appear to have been 
affected, as described by Mr Coyne. 


Because it was so noticeable at the branch, and the Peak is 
concerned with a software error rather than any other cause, I 
would expect any discrepancies in branch accounts to have been 
corrected.” 

(emphasis added) 


This is therefore undoubtedly a bug, but findings in respect of this 
therefore depend upon the nature of its impact on branch accounts. 
Dr Worden in his cross-examination changed the text of one of his 
entries in one of the appendices to his report, where he had said 
“This is further evidence of the failed recovery report doing its job - 
alerting [Fujitsu] to failed recoveries, so they can investigate them 
and make any necessary corrections to accounts.” He said this later 
part should be changed: “Perhaps I should have said have "guided 
the transactions to the correct state". He said it was caused by a 
complex grey communications failure, by which he meant an 
intermittent one, although he was not really sure where he had 
obtained the word “grey” from, he thought perhaps from a PEAK or 
KEL. 


The first PEAK PC0033128 related to an issue in November 1999 at 
Dungannon. The SPM reported a £43,000 discrepancy after 
balancing stock units and doing an office snapshot. An office 
snapshot is a Horizon report that showed the current accounting 
position of the branch, including any cash discrepancy. To produce 
the office snapshot, Horizon scanned the messagestore for the 
necessary information (eg. initial cash and stock levels, all cash and 
stock transactions, plus other service transactions). It then 
compiled that information together in order to produce the office 
snapshot. This is the “data tree”. On occasion Horizon would fail to 
read the necessary transaction data for various reasons so the 
report would be incomplete. This failure at Dungannon was caused 
by a missing payments node. 
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Two other branches were affected by the same issue, Yate Sodbury 
and Appleby-in-Westmorland. The snapshot at the former was 
incorrect showing a shortfall of £52,814.29 and Appleby’s snapshot 
showed a shortfall of £9,368.40. MSU were involved in the 
investigation of this and the Post Office submits that “it is therefore 
likely that the issues at Yate Sodbury and Appleby were resolved 
via a BIMS report and that the Subpostmasters were held 
harmless. However, due to the age of [this issue], comprehensive 
records are not available and therefore Post Office is not in a 
position to provide detailed commentary.” Two points need to be 
made. The shortfall was in a snapshot and not in a Branch Trading 
Statement; and secondly, there is no evidence of how it was 
corrected if at all. There are references in the PEAK in respect of 
Dungannon that shows the SPM and the Post Office agreed to 
remove the discrepancy from the account. For present purposes, 
although all these shortfalls are substantial, there is no evidence 
that the Post Office required payment of these sums from SPMs. 


In my judgment, the Post Office correctly identifies branch impact 
in the following passage in its submissions: 


“Of itself, Issue 1 [ie the issue above] would not affect branch 
accounts; there was no issue with the underlying transaction data 
and, if the office snapshot was re-run, it would very likely provide 
the correct information, because the data reading issue was 
temporary. However, if the branch ran the office snapshot, got an 
inaccurate report and then rolled over (making good any 
discrepancies in the process), then the shortfall would have an 
impact on branch accounts.” 


There is an associated issue in respect of data trees, namely PEAK 
PC0121925. As part of the changes made to move branch 
accounting from weekly cash accounts to monthly branch trading 
statements (a programme called IMPACT), changes to the data 
server were made to reduce the number of times that the 
messagestore was scanned to pick up transactions during 
balancing. A Riposte mechanism known as “Notifications” was 
used to add new transactions to the existing totals as further 
transactions were generated during the balancing process (rather 
than rebuilding the data server tree of transactions from scratch). 
One of the test branches experienced an issue, where the test 
branch had a gain of £45.05 following a cash declaration and 
rolling into branch trading. Initially, Escher were unable to 
replicate this scenario and so no further action could be taken. 
Subsequently, as demonstrated in a later PEAK in 2005, 
PC0123319 Fujitsu was able to replicate the scenario and 
implement a fix on 5 September 2005. The fix was implemented 
before any branches had switched to use the new branch trading 
code, meaning that the issue in PC0121925 could not have 
impacted any live branches. This is, however, a bug, error or 
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defect, but not in the Horizon System as it is used in actual 
branches. 


Entry 10 in the Bug Table therefore does relate to a bug in Legacy 
Horizon which had an impact upon branch accounts. As the Post 
Office submits, in this instance the impact of that bug was 
corrected. 


Two PEAKs relate to a second element of this, one in February 
2006, PC0132133 and another in March 2007, PC0144386. The 
Post Office accepts these PEAKs are essentially the same (one is 
said by the Post Office to be a manifestation of the other) and the 
former is said to “relate to an issue in which the notification 
mechanism referred to in PC0121925 was accidentally switched 
off”. There is no explanation of how or why this switching off 
occurred. The KEL that goes with these states in its title: 

“Title: Multiple cash declarations may cause incorrect figures in 
Discrepancy, Variance and Balance Reports 

Summary: Intermittent misleading figures in Discrepancy, Variance 
and Balance Reports” 


In the text in the KEL, certain passages make it clear that this is a 
software issue that persisted into 2007. 


“Solution - ATOS 


<b>Helpdesk:</b><br/>The Declare Cash problem clears itself 
overnight. If the PM logs a call on the day he is having problems, 
ask him to try the following workaround:<br/><br/>1. The clerk 
should log out of the affected counter.<br/>2. Another clerk 
attached to a different (individual, not shared) stock unit should log 
into the <b>same</b> counter, declare cash for his own stock 
unit, then logout.<br/>3. The first clerk can now login to the same 
counter and declare cash again. The variance should be correctly 
recalculated. 


Alternatively log on to a different counter and do the cash 
declaration there. If the workaround is not successful or the 
problem does not clear itself overnight, send a call to SSC, 
otherwise no call is needed. November 2007: a_fix is currently 


being piloted and is likely to be sent to the whole estate in January 


(COUNTER EPOSS 39 3 or later). If this problem is reported after 
COUNTER EPOSS 39 3 has been applied, send call to SSC.” 
(emphasis added) 


A fix is only required if there is a bug, error or defect in the 
software. 


246. 


247. 


The Post Office submitted that there was “no long term impact 
upon branch accounts”. This was therefore a slightly different 
approach to “lasting impact” upon branch accounts. “Long term” is 
not defined. If the impact of this bug occurred prior to a BTS then 
correction by means of human intervention and TC would be 
required. 


I therefore conclude that this is a bug with the potential to cause 
lasting impact to branch accounts. The fact that the impacts of its 
occurrence were corrected by means of TCs does not affect its 
existence. The November 2007 entry in the KEL shows that a fix 
was required in late 2007 intended to be released in January (which 
must mean January 2008). Accordingly, it cannot be considered to 
have been limited in its effect only to the early days of Legacy 
Horizon and obviously persisted for some years. 


11. Girobank discrepancies. 
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This was a Legacy Horizon issue that occurred between May and 
September 2000. Mr Coyne considered this was a bug. Dr Worden 
did not and stated that “the first fault concerns reports. A fault in a 
report is not a discrepancy in branch accounts, and only causes one 
if it causes a person to make a mistake.” This is now essentially 
accepted as a bug by the Post Office, but it is submitted it had no 
branch impact. It is included in paragraph 5 of Appendix 2 under 
the heading “the following bugs had no branch impact.” The 
detailed part of Appendix 2 dealing with this submits that there is 
no evidence of any financial impact upon branch accounts, let alone 
a lasting impact. This was in the early days of Legacy Horizon. 
There are said by the Post Office to be six distinct issues arising 
under this heading. 


I do not know if what are called “giros” are still in use at all in 
2019, but they look similar to cheques and used to be widely used, 
for benefits payments amongst other things. In basic terms, the 
giro (which is paper, and was sometimes included in a book of 
giros) would be exchanged for cash. A customer would present a 
giro for (say) £50 to the branch Post Office. The SPM or their 
assistant would take the giro, giving the customer the £50 in cash 
from the till, entering that transaction on Horizon. This would 
reduce the cash holding in the branch by £50. The giro would be 
sent to Girobank at the end of the day and that would be 
considered by Girobank together with the entries that would have 
come via Horizon. Girobank would then pay the Post Office the sum 
of £50 in respect of that particular transaction. 


If there was a difference between the value of the giros entered on 
Horizon, and the physical giros received by Girobank, then 
Girobank would not pay the Post Office the full amount. In other 
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words, there could be a mismatch between Girobank’s view of the 
amount of money paid out in exchange for giros (based on the 
physical giros it received), and the Post Office’s view (and hence 
the amount of money paid out of the branch by the SPM to 
customers presenting giros). 


The summaries of the different issues taken from the PEAKs show 
the following problems: 


1. Girobank taking the view that there was a discrepancy between 
giros received from (say) Branch X, and cash paid out by Branch X 
for those giros. The explanation for why this occurred was the “cut 
off” time for post being collected from Branch X (which would 
include the giros to go to Girobank) being earlier than the end of 
the trading day. Branch X might pay out (say) £85 to a customer at 
4.45pm, but the post (including all giro vouchers then available at 
the branch) might have left at 3.15pm or 4.30pm. This would lead 
to the difference explained at [250] above. It would also lead to an 
error notice being issued by Girobank. The Post Office’s 
submissions state that “in this particular case, the only possible 
impact would be if the branch had accepted the error notice 
received because of the reporting issue”. In other words, the only 
way the system would work correctly would be if the SPM did not 
accept the error notice. The Post Office have to rely upon a SPM 
not accepting an error notice issued to him or her to avoid any 
impact on branch accounts. That is not a software bug, it is true; it 
is however an error or defect in the system. The error notices are 
issued through or as part of the Horizon System. This type of 
incident is referred to in PC0044232, which concerned a £505.72 
discrepancy. This was a known issue dealt with by KEL 
MWright531p. This KEL is now said by Fujitsu to have been 
deleted and irretrievable due to its age. 


2. A second issue was identified by Fujitsu in the course of 
investigation of the incident in PC0044232. This was that the same 
£81 giro deposit had been included on two consecutive daily 
reports. Part of the Post Office’s submissions on this are as follows: 
“This is because the transaction was entered onto Horizon in a 
precise (and very small) window of time between two system calls 
being undertaken, resulting in a duplication. The overall branch 
position would still have been correct, but the daily reports to 
Girobank may have been wrong. If they were (i.e. if the same 
transaction was included on two consecutive daily reports), it is 
expected that this would have been spotted and a TC would not 
have been issued to the branch.” This is, in my judgment, evidence 
of a bug, error or defect in the system, as the duplication referred 
to should not have occurred. There is no evidential basis for any 
“expectation” that if the reports to Girobank had been wrong, it 
would “have been spotted” and led to the issuing of a TC. No such 
TC has been produced. 
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Some other issues relating to Girobank were identified by the Post 
Office separately, as issues 3, 4, 5 and 6 under the same heading of 
entry 8 in the Bug Table. Issue 3 applied to two PEAKs, both from 
2000. They were PC0052575 (in which the SPM reported 
discrepancies of £20 and £628.25) and the issue was diagnosed as 
arising out of the use of a shared stock unit. The Post Office 
submissions were “There is a window of time between a user 
printing and cutting-off a report. If another user was to perform a 
transaction during that window, that transaction may not show on 
the report. The issue was already due to be fixed in a future 
release”. Mr Coyne accepted in cross-examination that these were 
indications of a discrepancy being identified, but these were not of 
themselves evidence of a bug in Horizon. His overall evidence on 
this was contained in his final answer which was “It created a 
financial discrepancy within the Horizon system which could then 
ultimately have an impact on branch accounts.” I accept that this 
issue caused a financial discrepancy. Given the issue was “due to be 
fixed” in what is described as a “future release” the issue arose 
from the operation of the system and is therefore, in my judgment, 
correctly described as a defect. 


Issue 4 was identified as follows in the Post Office’s detailed 
submissions on Entry 8 in Appendix 2: 


“26. Issue 4 applies to PC0068633 (referred to at para 3.124). This 
Peak was raised in 2001 and so comprehensive records are not 
available. 


27. In that Peak, the Subpostmaster reported that his cash account 
showed two giro deposits of £1,503 but that his reports showed 
only one. The Subpostmaster received an error notice from 
Girobank which cleared the error, but he raised the issue because 
he believed that an error in Horizon was duplicating the 
transaction. 


28. The issue was caused by a cut-off being performed on one 
counter despite an attempt to print a transaction failing on another 
counter. This resulted in the cut-off report including the 
transaction that had failed to print. 


29. A fix was actioned by Fujitsu on 18 December 2001. 


30. Issue 4 only occurred in a very specific set of circumstances 
and would have had no direct financial impact on accounts; it 


merely had the effect that a transaction was missing from the 
reports.” 
(emphasis added) 


In my judgment, the content of those submissions clearly 
demonstrate that this was a bug, error or defect which had an 
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effect upon the report available to the SPM. This is made very clear 
in the PEAK itself, which states “I have duplicated this bug. In fact 
it occurs in all reports that use dataserver (i.e. the majority). I shall 
now check to see whether or not the problem still occurs at S10.” 
(emphasis added) The author of that PEAK was a Fujitsu employee 
and expressly referred to a bug. It did not have an effect upon the 
branch accounts of the SPM in question but it is undoubtedly 
evidence of a bug. 


Issue 5 applies to PC0073855 and PC0075312 from 2002. Again, 
there was not a great amount of material available due to the age 
of these PEAKs. A summary is as follows. 


In PC0073855, a SPM reported that the office snapshot figures 
were double the figures on the balance snapshot (with a £6.76 
discrepancy). Fujitsu were unable to replicate the issue and were 
therefore unable to issue a specific fix. However, a new version of 
the component was released with extra tracing code so that if the 
issue re-occurred, Fujitsu could gather more evidence. The SPM 
would have had an inaccurate report but the correct data was still 
available and this would not have affected the branch when 
balancing. In PC0075312, another SPM raised an issue with 
printing her giro deposits. The issue was identified as being 
caused by the same root issue as PC0073855 which was already 
with the development team. These did not impact branch accounts. 
However - and I consider this to be important - they constitute 
evidence of a bug. The KEL with which they are associated is KEL 
AChambers4410R, the same KEL as the PEAK in Issue 4. The only 
reason the Fujitsu “development team” would be involved would be 
to develop a fix. A fix is only required if changes are required to the 
software, which means there is a bug. 


Finally, Issue 6 applies to PC0076065, which was that two giro 
deposits were reported by a SPM not to be showing on the previous 
or that day’s reports. The SPM identified the amounts of the 
deposits (£11 and £24) which were said to be missing, but Fujitsu 
discovered that SPM had produced two reports, and on this 
occasion it was the SPM who was in error. One report did not have 
the deposits on it, but the next day’s report had both. I find that 
this is not a bug and is an example of user error (that is, and to be 
clear, SPM error). 


There is an associated issue with giros which is identified in 
another PEAK, which is not included in the six issues. I shall 
reproduce the Post Office’s submissions on this PEAK: 


“The issue in PC0050418 was thought to be the same issue [as in 
[251](1) above] - the branch’s largest discrepancy was £323.32. 
However, due to the length of time it took for the issue to reach 
SSC, the branch’s messagestore had been archived - the 
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Subpostmaster raised the call on 29 June and the call was sent to 
PINICL on 17 July. The Subpostmaster was told that they would 
need to provide further information (such as the Transaction ID) to 
allow Fujitsu to investigate. The Subpostmaster does not appear to 
have pursued this. The Peak notes that Girobank were going to 
send an error notice, but due to the age of this Peak the relevant 
records are not available and Post Office is not in a position to 
provide detailed commentary.” 

(emphasis added) 


In my judgment, that is prima facie evidence of a shortfall in the 
sum of £323.32 being caused to that branch’s accounts as a result 
of this defect or error, notwithstanding that the SPM reported it. 
The SPM was told further evidence was needed in order to 
investigate. It is a self-contained example that in my judgment 
supports the claimants’ case on the existence of a bug, error or 
defect affecting Girobank issues. The fact that the SPM for 
whatever reason did not pursue it does not affect that. 


Although the very numerous PEAKs identified in respect of this bug 
in the Bug Table share some common features, as can be seen 
above there are different issues in respect of many of the PEAKs. 
This was counted as a single bug in the Bug Table, even though my 
detailed findings above show that there were more than one bug, 
error and defect that related to Girobank. I will however for 
consistency count it as a single bug in the overall total, as that is 
how the experts treated the total number of bugs in the Bug Table. 


12. Counter-replacement issues. 
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This also is a Legacy Horizon issue. The counter is part of the 
hardware of Horizon. 


There were two KELs associated with this dealt within the bug 
table. The first was created in 2000 and last updated in July 2007, 
and refers to 88 PEAKs. The second was created in 2002 and notes 
occurrences running up to 2009. It is now accepted as a bug by the 
Post Office in paragraph 6 of Appendix 2, but is said to have had 
transient impact. Mr Coyne considered that when replacing a 
counter within a branch, the process could result in “the total loss 
of a transaction”. Dr Worden stated that the cause, recorded in the 
first KEL (which was created in December 2000 and last updated in 
July 2007) was that Riposte was coming online from the Recovery 
mode too early, and causing messages to be overwritten. 


In theory, when a counter was replaced, it builds its messagestore 
by replicating with its neighbours in “recovery mode”. The 
neighbours it has depends on the office size (which would affect the 
number of other counters) and node number. For a single counter 
office, the neighbours are the correspondence server in the 
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datacentre and the mirror disk (the second hard drive in the same 
counter). For a multi-counter office, the neighbours are the 
correspondence server and all other nodes at the office, or all the 
other nodes in the office (known as slaves) depending upon the 
node number of the counter being replaced. 


A replacement counter is supposed to come out of recovery mode 
when it believes it has successfully replicated all relevant messages 
from its neighbours. The Post Office submissions state that “In this 
case, the replacement counter came out of recovery mode early, 
before it had replicated all messages from its neighbour. The 
replacement counter started writing messages from the point at 
which it believed it had replicated all relevant messages from its 
neighbour. This meant that it used message IDs that had been 
used for messages that had not been replicated from its neighbour 
and this prevented the “missing” messages from being replicated 
later on (because that would have created duplicate message IDs). 
The missing message was therefore “overwritten” by the 
replacement counter.” 


The Post Office also stated that “the issue arose in cases of counter 
replacements where the new counter was not connected to all of its 
configured neighbours while rebuilding. This may have been 
because the branch infrastructure was not complete (eg not all 
neighbouring counters are online, multiple swaps or a counter 
increase/decrease occurring) or the engineer did not connect the 
system properly.” Regardless of the reason, the failure to replicate 
messages is a failure in the internal processes of Horizon to cope 
with the hardware replacement of the counter, and in my judgment 
is a bug, error or defect. 


PEAK PC0052823 gives a technical explanation for this, part of 
which is: 


“It would appear that after the recovery from the squirrel 
completed, the message processor came out of recovery mode after 
synchronising up to message number 660 for node id 1. Suspect 
this was by replication with the Correspondence server. Counter 
then wrote a Riposte on-line message as 661 for node 1 before 1 
second later attempting to synchronise to the F: drive mirror 
message store. At this point a red event regarding 'self originated 
message’ was generated, the server switched back to recovery 
mode and the remaining messages from the F: drive mirror 
message store above 661 were synchronised.” 


It also notes that “Gareth Jenkins viewed this error on rig with 
Mike Berrisford.” 


The nature of the correction in the KEL was stated as being “to find 
the overwritten transactions for reconciliation we need to look at 
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the Ripostemirror messagestore' followed by detailed instructions”. 
Riposte, as has been explained, was part of Legacy Horizon and 
was a product provided by Escher, but it is plainly part of Horizon. 
Dr Worden also stated that “the incident arose from a hardware 
replacement (probably from a hardware fault) not from a fault in 
Horizon. It is a different kind of recovery issue.” In my judgment 
the hardware is part of Horizon. 


There was a short term and long term fix. The former dealt with the 
actual branch by the SSC inspecting the Riposte mirror 
messagestore, and retrieving the specific messages which had been 
overwritten. Information of the overwritten messages was passed 
to MSU who created a BIMS report for the Post Office. The Post 
Office submits that “an error notice would have been issued to hold 
the branch harmless thereafter”; no such error notice has been 
produced but I consider that the KEL and PEAK together suggest 
that such a notice was produced in order to correct the impact 
upon branch accounts. 


The long-term fix for the issue was an actual fix, which is detailed 
in PC0052823. This involved enforcing a minimum number of local 
neighbours for replication and then to slowly lower this number 
over time. A further change was also made to stop Riposte writing 
messages as it came online. This further change was a change to 
how Riposte operated. This therefore means that the fix changed 
the software of the system. 


The dates of other PEAKs shows that this persisted. 


PC0071836 is accepted by the Post Office in Appendix 2 as an 
example of the same issue as KEL JBallantyne5328R; this is dated 
November 2001. This SPM had a receipts and payments mismatch 
of £3.27 as a result of three overwritten messages 

following a replacement of the branch’s single counter. The same 
fix was applied following KEL JBallantyne5328R. PC0133822 is not 
the same issue as JBallantyne5328R but is accepted as being 
related. This PEAK is dated 24 March 2006. The branch had two 
counters removed, leaving it as a single counter branch. However, 
the counter did not have a mirror disk; the mirror disk was a 
second hard disk within a single counter that provided a replication 
neighbour for the main hard disk messagestore. This meant that 
the branch would have no replication of data if it was not 
connected to the datacentre and six messages on the counter had 
not been replicated to the data centre. The messages were 
extracted and sent to MSU for a BIMS report to be raised. The 
Post Office submits that “An error notice would have followed the 
BIMS report and so there would have been no lasting impact on 
branch accounts”. However, that effectively accepts that absent the 
error notice, the branch accounts would have been affected. 
PC0153851 is not the same issue as JBallantyne5328R but it does 


272. 


involve a receipts and payments mismatch; it is dated 7 February 
2008. Riposte failed to index four messages resulting in some 
items being missing from the receipts side of the balance report. 
The PEAK itself notes that the branch did not experience a 
discrepancy as a result because this was a reporting issue only; 
indexes are not used when replicating data and so cash/stock were 
unaffected. This does however show that the problem with Riposte 
persisted. 


I find that this was a bug, error or defect with potential lasting 
impact to branch accounts. The fact that it originated with the 
necessity to replace a counter (which is plainly hardware, as Dr 
Worden notes) does not matter because the hardware is part of the 
Horizon System. Nor does the fact that the impacts of its 
occurrence were corrected by means of error notices/TCs affect the 
existence of the fault. A change, or changes, had to be made to 
Riposte to stop this occurring, and there were still later 
manifestations of it occurring. 


13. Withdrawn stock discrepancies. 
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This is a Horizon Online issue. The Post Office does not accept 
these as a bug. Mr Coyne maintains that it is, and in the bug table 
he extracted part of a PEAK that stated these “Can cause confusion 
and unexpected (though hopefully temporary) discrepancies at 
branches by allowing them to declare stock which has already been 
withdrawn.” Dr Worden stated that “some impact on branch 
accounts cannot be ruled out, although it is small”. The Post 
Office’s detailed submissions in Appendix 2 of its Closing 
Submissions concentrated on the fact that when the Post Office 
withdrew stock - which of course is part of the way that the Post 
Office manages its business, adding and removing types of 
products from time to time - this removal of products is done by 
means of an update to reference data, “and not a change to the 
core code in the system.” This again is concentrating on the code, 
rather than the way the system operates. 


There were two steps that led to this happening. The Post Office 
submissions on this rather gloss over the entries in the PEAK, so I 
will reproduce the actual entries of 19 January 2011 in PEAK 
PC0207834. Although that is the date of PEAK, the text makes 
clear that the event occurred in November 2010. The context is 
that the Post Office withdrew the £5 saving stamp from its 
business, but obviously some branches retained (or had at the time 
they were withdrawn) such stamps in their stock: 


“This office physically held 137 £5 PO saving stamps and did not 
rem them out before the date the rem out icon disappeared. The 
office physically returned the stamps to Transaction Processing as 
advised and the office then did a Trading Period balance on 
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17/11/2010 and showed this value as a loss. The spmr put the cash 
into the till at this point to make good the loss. Transaction 
Processing then issued a transaction correction to the branch on 
19/11/2010 for the value of £685 to make good the loss. The office 
brought this to account and when they did the next Trading Period 
balance on 15/12/2010 it showed a gain of £537.49 which was 
settled centrally. The spmr says that they did have a loss showing 
again in the balance for the value of the PO Saving stamps, £685, 
again, even though this stock line was not showing in either adjust 
or declare stock. After this Trading Period balance the spmr took 
the £685 back out of the till and when they balanced again on 
12/01/2011 it again showed a loss of 137 £5 PO Saving stamps and 
the nett discrepancy in the branch was £1370, which again was 
settled centrally. The office declares stock when they do the 
balance but insist there is no line showing in either adjust stock or 
declare stock for the PO Saving stamps. The spmr is convinced the 
£1370 loss, which is twice the £685, is due to the fact these stamps 
are still on the system somehow and keep giving the office the 
losses. The office has faxed a copy of their stock holdings for the 
branch at the moment and it clearly shows “Saving Stamps £5 
137”. This stock line should have been removed when the stock 
item was made obsolete and removed from Horizon. 


The two user names who do the balance are SKIOO1 and PCAOO1. 


I have spoken to Phil Herrett in Transaction Processing who has 
confirmed he is aware of about 8 offices with similar issues with the 
stamps still showing on the stock. He gave me FAD 0640123 as an 
example. 


Can this be investigated to see why the stock is still showing in the 
office, and if this keeps giving the office a loss of £685 every time 
they do a Trading Period balance.” 

(emphasis added) 


The Post Office submissions state that “However, the 
Subpostmaster in this case returned the stamps (with a total value 
of £685) to Post Office without first remming them out. This aspect 
of PC0207834 was pure user error.” However, that is not what the 
first entry in the PEAK states. It states that the SPM could not rem 
them out because the rem out icon had disappeared for this stock. 
That is undoubtedly an error by the SPM. However, the fact that 
the withdrawn stock remained on the stock holdings, which are 
part of the branch accounts, even after the stock has been returned 
and a TC issued to correct the discrepancy for their value, cannot 
be laid at the door of the SPM. Further, other offices (“about 8” of 
them) were also experiencing the same issue. This is an error or 
defect with Horizon in my judgment. Part of the submissions by the 
Post Office states as follows: 
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“8. The branch declared a £685 shortfall. The lack of a rem out 
meant that Horizon thought that the branch was still holding the 
stamps and therefore when the Subpostmaster either declared or 
adjusted the stock of stamps to zero without a corresponding gain 
in cash, a shortfall was generated. The Subpostmaster elected to 
make good the shortfall and a credit transaction correction was 
subsequently issued for £685 to rectify the issue. 


9. However, a bug in Horizon caused the £685 of stamps to be 
subsequently re-introduced into the branch’s accounts on two 
occasions. By this point, Horizon was showing that the branch was 
holding £1,370 of the stamps. The Subpostmaster noticed this, 
adjusted or declared the stock as zero, and reported the issue.” 
(emphasis added) 


The Post Office therefore accept in those submissions that the 
subsequent effect upon the accounts was a bug in Horizon. I do not 
consider it much matters whether this is characterised as a bug, or 
an error, or a defect in Horizon. It plainly was a bug that had the 
potential to cause lasting impact to branch accounts (and in this 
case actually did so prior to the issue of the TC). 


14. Bureau discrepancies. 
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These relate to foreign exchange, hence the name. It must be 
differentiated from alleged bug at entry 23 in the Bug Table (also 
concerning foreign currency, but entitled Bureau de Change). This 
is a Horizon Online issue and arose in 2017. 


Mr Coyne considered it to be a bug, and Dr Worden effectively 
agreed in that his entry stated “This appears to be a system error 
with impact on branch accounts. Although it is possible that a 
subsequent discrepancy between branch accounting and POLSAP 
would reveal the problem, leading to a correction (e.g. see Peak 
PC0265443, and Mr Coyne's para 3.146), I cannot be certain of 
this.” It is now accepted by the Post Office in paragraph 7 of 
Appendix 2 as one of a number of “bugs with lasting impact 
(although they were resolved)”. The detailed part of Appendix 2 
states that Mr Coyne has drawn together two distinct issues. 
Paragraph 4 of the detailed submissions also states the following: 


“Bug 14: Bureau Discrepancies is a bug with the potential for 
lasting financial impact. There are two distinct issues which fall 
under this heading. With regards to the first issue the branch was 
made good and a fix was implemented. The second issue was not a 
bug in Horizon nor an issue which could have impacted branch 
accounts; it created what was essentially a cash flow problem for 
the branch.” 
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So far as the first issue of the two identified by the Post Office is 
concerned, given a software fix was required, it is undoubtedly a 
bug, and it is correct to record (as the Post Office do) that there 
was the potential for lasting financial impact, as without that, there 
would be no need for the “branch to be made good” because that 
means the Post Office had to correct the branch account 
discrepancy caused by the bug. In a sense that is stating the 
obvious. Given those submissions, I will deal with the first issue 
very briefly. 


This occurred as follows. In August 2017 an SPM tried to pre-order 
£1,000.07 in Indonesian rupiah and £204.59 in Singaporean dollars 
for a single customer. The rupiah order was created, but there was 
a network timeout at the point when the SPM tried to perform the 
dollar order. When the system re-connected, a warning message 
suggested that the second order may not have been placed, but the 
basket and transaction log were showing both orders. The SPM 
attempted to cancel the whole order, but the cancellation only 
worked for the rupiah order, leaving the dollar order of £204.59 in 
the branch accounts. This is because as recently as August 2017 
multiple currency orders were processed as multiple transactions. 
The rupiah order was added first at the counter and then sent to 
the BAL, and this caused the order ID PBX1048411 to be created. 
The network timeout occurred at this point - such a timeout has 
nothing to do with the SPM. At that time that the dollar order had 
been added to the counter stack but it had not yet reached the 
BAL. This meant that it did not become associated with order 
PBX1048411 and so, when the SPM attempted to cancel the whole 
transaction (PBX1048411), the dollar order was not cancelled. This 
then created a shortfall as Horizon will have expected the SPM to 
have taken a payment from the customer of £204.59. 


This issue could only occur because of the very specific 
circumstances of PC0261541 including the occurrence of a network 
timeout at a very precise moment in the transaction. The Post 
Office’s detailed submissions state the following: 


“The PEAK notes that the issue was referred to Post Office in order 
for them to “decide what reconciliation or transaction correction is 
required to balance” at the branch. Post Office have confirmed 
that a transaction correction was issued to the branch on 2 
November 2017 and accepted by the branch on 7 November 2017 - 
this is not apparent from the PEAK itself.” 


This means that the accounts of that branch were incorrect, and in 
the Post Office’s favour, for the period mid-August to mid- 
November 2017. A change to the software was made in November 
2017 via a change to the AP-ADC script which had the effect of 
simplifying multiple currency orders to be processed as a single 
transaction. 


283. 


284. 


285. 


286. 


The second issue is even more marked, in my judgment, in terms of 
arriving at the correct answer to Horizon Issue 1. PC0265443 is 
dated 29 December 2017 but relates to a call on 17 December 
2017. The Post Office’s internal cash management team’s system 
(which is referred to as POLSAP) indicated that the branch in 
Burgess Hill was holding €4,500 and US$1,000 more than the 
branch was declaring it held. This meant that, from the cash 
management team’s perspective, the branch held sufficient cash to 
conduct transactions, and so were reluctant to provide more cash 
to the branch. This resulted in a cash flow issue for the branch 
who were sometimes unable to serve customers as a result. This 
PEAK is 57 pages long; this is many pages longer than most PEAKs, 
which are usually 10 pages or less. It evidences a very long 
investigation. 


This investigation is best summarised by quoting the Post Office’s 
own submissions under the heading “Resolution”: 


“13. The Peak is long and complex and it demonstrates that the 
issue was extensively investigated by Post Office, ATOS, Accenture 
and Fujitsu, since the issue was thought to be occurring 
somewhere between the branch, Horizon and POLSAP. The 
conclusion of Fujitsu and Accenture’s investigations (over 
approximately six months) was that the issue was not caused by 
any issue within their respective systems. Further, Post Office sent 
a trainer to the branch to verify the cash position and found no 


discrepancies; the amount of cash physically held in branch was 
the same as the amount showing on Horizon (but not POLSAP). 


14. The root cause of the issue does not appear to have been 
determined. As it did not appear to be a case of user error, it 
appears that Post Office wrote off the discrepancy in POLSAP at no 
cost to the Subpostmaster to bring that system’s figures into line 
with those in Horizon.” 

(emphasis added) 


This shows that POLSAP was not in line with Horizon; that no 
problem could be found by Fujitsu with its system, nor by 
Accenture; but that the amount of cash (in terms of foreign 
currency) in the branch was being correctly declared by the 
branch. 


Not only do I find that this is evidence that supports the existence 
of this bug in the Bug Table, which potentially causes lasting 
impact to branch accounts, but it is also consistent with my finding 
on a separate issue, namely the need to use ARQ data in 
circumstances where there are serious disputes between the Post 
Office and SPMs, rather than Post Office management data. 


15. Phantom Transactions. 
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These arise under Legacy Horizon in terms of the dates. There are 
three different issues grouped under this heading. The Post Office 
does not accept these as a bug. Mr Coyne in his cross-examination 
stated that this was referable to Horizon Issue 4, rather than 
Horizon Issue 1. He also accepted that some of the issues were not 
bugs in Horizon. Dr Worden relied upon the fact that what he 
called “the master PEAK” was closed as “no fault in product”; 
however, the evidence of fact on this is dealt with in the 
substantive judgment where I consider the evidence of Mrs Van 
Den Bogerd when the PEAK was put to her. 


Dr Worden stated that “There is no evidence for bugs in Horizon 
with impact on branch accounts.” That entry was obviously before 
the cross-examination of Mrs Van Den Bogerd, which in my 
judgment provided greater factual information. I do not consider 
reliance can be placed upon the Fujitsu conclusion in the PEAK. 


PEAK 0065021 is dated 17 April 2001. I consider it to be an 
important PEAK. It relates to multiple branches. It concerns 
phantom transactions. It identifies dissatisfaction from more than 
one SPM as to how phantom transactions are being investigated 
and resolved, or more accurately, how they are not being. It shows 
calls being “closed” - ie brought to an end, without the permission 
of the SPM, even though that should not be done. It also shows at 
least one SPM threatening not to comply with their contractual 
obligations due to lack of confidence in the system, and also threats 
of legal action. Further, in one branch, where items were the 
subject of phantom transactions (according to the SPM) ROMEC, 
the Royal Mail’s own engineers, attended that branch to fit 
suppressors and other equipment in an effort to rectify this. 


The PEAK plainly records the involvement of ROMEC, the Royal 
Mail’s own engineering personnel, as follows. “ROMEC have been 
to site and state that they have_actually seen the phantom 
transactions, so it is not just the PM's word now.” (emphasis 
added). I consider that this is significant. Mrs Van Den Bogerd 
agreed that this was “independent site visit corroboration of the 
problem by Royal Mail’s own engineers at the branch”, and she 
also agreed that this was “clearly not user error any more”. This 
entry in the PEAK is, in my judgment, important corroboration from 
those with experience of Horizon (the Royal Mail’s own engineers) 
who state they have seen the phantom transactions. The conclusion 
reached by Fujitsu and recorded in the PEAK was as follows: 


"Phantom transactions have not been proven in circumstances 
which preclude user error. In all cases where these have occurred 
a user error related cause can be attributed to the phenomenon." 
The PEAK also concludes “No fault in product". 
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I reject both of those conclusions. In my judgment, they are both 
self-serving and are contradicted by the factual matters reported 
within the PEAK by Fujitsu, which include that these have been 
witnessed by ROMEC engineers. 


A Fujitsu entry from 19 June 2001 in PEAK PC0065021 states: 
“I now have pressing evidence to suggest that unwanted peripheral 


input is occurring, the likely source being the screen. This has been 
seen at Old Iselworth (OI) and Wawne (W) with OI being the best 


site; when the PM has been asked to leave the screen on overnight 
I have observed system activity corresponding to screen presses 
happening with no corresponding evidence of either routine system 
activity or human interference, the way forward now is to correlate 
this with the Microtouch supplied monitoring software and to this 
ends Wendy is arranging for installation of Kit at OI on Friday, we 
can then, provided the PM agrees, leave screens on over the 
weekend and record what happens. 


Once these results have been analysed I feel sure that we will be in 
a position to move forwards at OI. All other cases should be 
considered on their individual merits but you must appreciate that 
this is a fairly intensive analytical activity and I cannot hope to 
provide answers on all cases in the short term.” 


I consider this to be evidence of a bug in Horizon. Mr Coyne notes 
that there is no specific branch account discrepancies noted in this 
respect; however, given Fujitsu concluded in November 2001 the 
following “Phantom Txns have not been proven in circumstances 
which preclude user error. In all cases where these have occurred 
a user error related cause can be attributed to the phenomenon. I 
am therefore closing this call as no fault in product” then the lack 
of noted discrepancies for this is not surprising. Nor, in my 
judgment, is it necessary for there to be specific identified 
discrepancies given the word “potential” in Horizon Issue 1. 


Both the ROMEC engineers who had observed the phantom 
transactions, and indeed Mr Carroll in his entry of 19 June 2001, 
considered based on the evidence they had at that time that there 
were so-called phantom transactions occurring. Mr Carroll’s entry 
in particular that states “when the PM has been asked to leave the 
screen on overnight I have observed system activity corresponding 
to screen presses happening with no corresponding evidence of 
either routine system activity or human interference” can only 
sensibly be explained by a bug. The Post Office submitted that “For 
the reasons set out above [in its submissions], there is no indication 
that phantom transactions had a cause other than user error, as 
indicated in the PEAKs.” However, that ignores the entries in what 
was referred to as the master PEAK that indicate precisely the 
opposite. 
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I consider that an inference, that is to say a common sense 
conclusion, can be drawn from all the evidence on these matters 
that there was such a bug in 2001 that caused phantom 
transactions. This had the potential to cause impact to branch 
accounts. 


16. Reconciliation issues. 


296. This heading in the Bug Table grouped both Legacy Horizon and 
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Horizon Online issues with reconciliation together. The Post Office 
does not accept these as a bug. There are a number of different 
issues grouped together in this heading. The issue was that the 
SPM was shown a discrepancy on his or her screen. Mr Coyne 
accepted that the discrepancy would not be shown in the branch 
accounts; Dr Worden stated that “as it concerns an issue in 
reporting, the software fault (which was fixed after 5 months) had 
no direct impact on branch accounts. The only effect of an error in 
this report would be to mislead or confuse the Subpostmaster - 
probably leading him to check his figures more carefully and 
costing him some time.” (emphasis added) 


That entry by Dr Worden plainly accepts a software fault; the fault 
was in fact a miscounting of the number of files by the system. 
However, there are six different PEAKs grouped under this heading 
and the process of reconciliation, which is effectively the 
comparison by Horizon of two different sets of data, is part of its 
design function. The recovery messages were held in the branch 
messagestore. Mr Coyne in his cross-examination also stated that 
this was referable to Horizon Issue 4, rather than Horizon Issue 1. 
Three of the six different issues were missed off the copy of the 
submissions lodged for oral submissions in July 2019, but this was 
made clear by the omission of “Issue 4” to “Issue 6” and following 
submission of Mr Parsons’ 19" witness statement these were 
added. 


This is one area where the cross-examination of Mr Coyne in 
particular does lead to a conclusion that this is not a bug that 
causes potential lasting impact to branch accounts. The six matters 
identified by the Post Office in the detailed part of Appendix 2 all 
deal with different PEAKs. 


Peak PC0039832 is dated 3 March 2000 and is an example of an 
issue affecting a SPM’s Cash Account Period. Reconciliation 
discrepancies appeared but did not feature on the expected 
reports. The counter calculated information each day for the Cash 
Account in two independent ways to ensure that they matched and 
a report was generated if they did not match. In this case, the 
reconciliation software detected discrepancies relating to two low 
value transactions (£8.06 and £0.08, totalling £8.14). The PEAK 
suggests that “there was a bug in the reconciliation software, 
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although the Peak is not fully conclusive” as the Post Office’s 
submissions recognise in this sentence that I have quoted. This 
would have generated a false reconciliation report but there would 
have been no financial impact on the branch. The reconciliation 
error was captured, as that is what triggered the investigation in 
the PEAK. The fix is documented in PC0047955 of 19 June 2000. It 
is evidence of a bug but given the nature of it, it would not have 
impacted upon branch accounts. 


PEAKs PC0075240, PC0075415 and PC0077508 are all from April 
2002. These all relate to the same (or at least a markedly similar) 
issue where a branch counter total differed from the amount on the 
TPS host. There was no discrepancy in branch accounts. In all 
these cases, the counter calculated a value of 1p and the 
calculations carried out at the host gave a value of £0.0099. As the 
first three digits were 000 the system took this to be Op and so 
reported a false discrepancy. There was however no financial 
discrepancy. A fix was however required, which demonstrates it 
was a bug, error or defect, but it was not one with the potential to 
affect branch accounts. 


PEAK PC0049578 is from July 2000. It was raised in testing and 
fixed before the software went live. It was a bug, error or defect 
but the purpose of testing is to discover such things, and as such 
this had no potential to impact branch accounts. The problem was 
producing a report used to confirm that all data has been passed to 
TIP. The underlying data transferred to TIP was correct, however, 
the number of files transferred did not match. There was also a 
report to confirm what files were transferred, and that report was 
also incorrect in the count of files transmitted. If this had gone to 
live then the answer to this might be different, however regardless 
of that, it was fixed before release. 


The fourth issue relates to a PEAK dated May 2000. The 
submissions stated that “the Peak describes an automatic detection 
of a receipts and payments mismatch of £4,464.46. It was 
automatically reported to Fujitsu and was investigated. The issue 
related to a hardware failure. It therefore did not relate to 
reconciliation issues arising as a result of a software fault.” It is 
however a defect or error with the Horizon System, which 
comprises both hardware and software. 


PEAK PC0236246 is dated 12 September 2014. It relates to a 
discrepancy of £110,706. The Fujitsu entry in the PEAK is 
somewhat illuminating: 


“This problem is likely to occur whenever products are introduced 
with non-midnight times, a common occurrence when data is 
created (and subsequently corrected) in MDM. Based on current 
data, such products may be seen roughly once a month. 
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When it does occur, any transactions against the products are 
omitted from the CTS report, resulting in discrepancies in trading 
totals, and operations are required to investigate, usually requiring 
the assistance of development. 


The CTS Report is used by POL to settle payments with their 
Clients._ Incorrect values on the CTS report cause Post Office 
Clients to lose confidence in POL Accounting resulting in loss of 
face to our Customer. POL also therefore have loss of confidence in 
the precision of our IT Systems and this is an especially sensitive 
issue whilst POL are being investigated for accounting accuracy. 


POL’'s lack of confidence in our IT Solution will not help our bid for 
the renewal of the Front Office contract.” 
(emphasis added) 


CTS stands for Client Transaction Summary. In my judgment, the 
error shown in this PEAK is plainly a software fault that affects the 
accuracy of accounting at the Post Office. However, in this case, it 
has an effect upon the CTS Report, used to account between the 
Post Office and its clients, and not branch accounts. There was 
therefore, under this heading, a software fault, however it had no 
potential to impact upon branch accounts for the reasons I have 
identified above. 


The final of the six different issues also relates to the CTS Report, 
but one of the two PEAKs is that which I have already reproduced 
at [303] already. The other is PEAK PC0204872 and is dated 4 
October 2010. The Post Office submissions state that “the issue 
relates to a difference between the CTS Report and the branch 
figure. The root cause identified was an issue with POLSAP, and not 
with Horizon”. The PEAK records that there was a problem with 
the CTS report for a branch concerning the Alliance & Leicester: 


“The CTS report is received daily and is compared with the vendor 
(in this case A&L) reports. The figures for each day should match. 


If the CTS report is larger than the vendor figure, the vendor 
account will be credited. The credit usually shows a couple of days 
later as a positive discrepancy. 


The CTS report was showing as being larger than the vendor 
figures on the following dates, although there does not appear to 
have been any counter credit showing on the vendor figures 
following on from this: 


7th May 2010 - CTS was greater than vendor figures by £84.86. 
POL have suggested that this may have been related to an event 
from 27th February for FAD 490519, although we can find no BIMS 
record of this from a Reconciliation perspective. 
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25th July 2010 - CTS was greater than the vendor figures by 
£3,260.00. No additional information is available. 


27th August 2010 - CTS was greater than the vendor figures by 
£846.00. No additional information is available.” 


The next entry is: 


“Following this additional information this would appear to be an 
issue with POLSAP. 


The SSC do not support POLSAP, so we can take no action here.” 


That submission and the entry in the PEAK is of assistance in 
dealing with the ARQ data/management data dispute, but does not 
necessarily evidence a bug in Horizon potentially impacting branch 
accounts. It does not, however, for obvious reasons, take the matter 
very much further forwards. 


I find that these PEAKs, although demonstrating software faults, do 
not show that there is a bug with the potential to cause impact to 
branch accounts 


17. Branch Customer discrepancies. 
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This was a Legacy Horizon issue and is an entry only in respect of 
Horizon Issue 4 and recorded as such in the Bug Table. The Post 
Office does not accept these as a bug. Mr Coyne accepted in his 
cross-examination that the entry in the PEAK did not suggest any 
impact upon branch accounts. The entry in the bug table originated 
in a single PEAK, although there were two PEAKs that relate to the 
same issue, which was a discrepancy between the financial records 
held by the Post Office (from Horizon) and a bank’s records, after a 
counter crashed whilst a transaction was being processed. The 
amount in question was £165.26. The date of the PEAK is 29 March 
2008 and the incident occurred between 22 March 2008 and 26 
March 2008 (the date of the card account). 


The SPM did not take the sum in question from the branch 
customer, but that person had their account debited. The first 
PEAK was raised by Fujitsu who picked up on this as part of 
automatic reporting (an NB102: Exception Summary report) and 
the SPM was advised how to go through recovery. Fujitsu had 
issued a BIMS Incident Report to the Post Office, and had 
confirmed that the recovery messages were in the branch 
messagestore (even though they do not appear to have been shown 
by Horizon to the SPM in the first place). However, the branch 
appeared in the NB102 report again, and upon investigation this 
was found to be because the SPM (quite properly, in my judgment) 
had not followed the recovery messages because they had not 
taken the sums from the customer. 
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This is effectively accepted by the Post Office in its submissions 
which state: “Fujitsu issued a second BIMS Incident Report to Post 
Office which advised that while the recovery messages were 
received by the SPM, the SPM had declined them. This indicates 
that no money changed hands during the transaction (i.e. the SPM 
elected not to recover the transaction because no money had 
changed hands; if money had changed hands, the SPM should have 
elected to recover the transaction).” 


The Post Office also submits the following: 


“While there was a possibility that the Financial Institution 
registered a withdrawal from the customer’s account (depending 
on how the Financial Institution’s IT system is configured), this 
would have caused a loss for the customer, not the branch 
(assuming that the customer received no cash from the branch). If 
the customer did receive cash from the branch, the resultant 
discrepancy would have been caused by the branch not following 
the recovery procedure correctly.” 


I find that a very surprising submission. The counter - part of 
Horizon - crashed during a transaction. That is not the fault of the 
customer, and it is not the fault of the SPM, nor is it the fault of the 
customer’s bank. I do not know if solace is taken by the Post Office 
that any loss caused to the branch customer can, apparently, be 
laid at the door of the customer’s branch, but I doubt solace can be 
properly taken. Nor do I see how the SPM can be criticised for “not 
following the recovery procedure correctly” as it is accepted that 
this should have occurred if the SPM had taken money from the 
customer, but if they had not, it should not have been followed. On 
either analysis, this is a fault with Horizon. 


However, it does not appear to have occurred more than this single 
occasion. I agree with Mr Coyne, and this is relevant to Horizon 
Issue 4 only. I consider that it is an error or defect in Horizon, but 
does not constitute one that had the potential to cause impact to 
branch accounts. 


18. Concurrent logins. 
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This is a Legacy Horizon bug. It is accepted by the Post Office in 
paragraph 7 of Appendix 2 as one of a number of “bugs with lasting 
impact (although they were resolved)”. In paragraph 2 of the 
detailed part of Appendix 2 dealing with this bug, the following is 
stated by the Post Office: “Post Office accepts that Bug 18: 
Concurrent Logins had a potentially lasting financial impact. There 
is no evidence of any discrepancy in the PEAKs referred to by the 
experts.” The problem was in the early days of Legacy Horizon 
when it was possible for users to log in to two terminals at once. Dr 
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Worden in the bug table made the following useful summary 
statement: 


“discrepancies could occur - manifesting themselves as a 
receipts/payments mismatch. This had the potential to affect 
branch accounts. The mismatch would bring it to the attention of 
the Subpostmaster, who would require it to be investigated, except 
possibly in the case of small mismatches, which he might pass off 
as an error in the branch (e.g. of counting stock).” 


He also stated that “....Fujitsu believed it was a problem with the 
underlying Riposte software, and passed it to Escher. In September 
2000, the problem was 'Now formally fixed in Build 223 update 19 
which was released overnight.’ However, the new release from 
Escher did not, as it was expected to, fix the problem. Escher 
denied that it was a bug in Riposte, but Fujitsu believed in July 
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2001 that ‘This is clearly a bug in the Supplier's code’. 
(emphasis added) 


Paragraph 2 of the detailed submissions by the Post Office in its 
Appendix 2 state: 


“[The] Post Office accepts that Bug 18: Concurrent Logins had a 
potentially lasting financial impact. There is no evidence of any 
discrepancy in the PEAKs referred to by the experts.” 


Dr Worden considered that “these faults had the potential to 
produce discrepancies in branch accounts, of small amounts, for a 
short period of time”. How the bug is therefore characterised 
depends upon my findings in relation to lasting and transient 
impact, which is dealt with in the substantive judgment to which 
this is an appendix. I find that this was a bug, with the potential to 
cause lasting impact to branch accounts. Dr Worden attempts to 
play this down and so, in my judgment, does the Post Office. Fujitsu 
clearly recognised and recorded that in 2001. The fact that the bug 
was in the code supplied by Escher, as part of Riposte, and not that 
of Fujitsu is wholly irrelevant for the purposes of the Horizon 
Issues. 


One of the PEAKs related to the printing of a report whilst a 
counter crashed. The Gateway counter was then used, and because 
the first counter had crashed (what is called a “slave” counter) this 
was permitted because it could not communicate to the Gateway 
counter who was logged in to it, or what it was doing. This PEAK is 
PC0027581 and relates to an incident in July 1999. That PEAK runs 
to February 2002. It is notable, and continuing the theme of 
Fujitsu’s closure codes, that it is closed with the code 
“Administrative Response” even though the entry for 13 July 2001 
states “This is clearly a bug in the Supplier’s Code and as you say, 
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management pressure must be brought to bear as necessary to 
make the Supplier accept and respond to that fact.” 


The second PEAK is PC0051327 dated 2 August 2000. The entry 
which opened the PEAK is: 


CALL PC0051327:Priority B:CallType L - Target 02/08/00 12:32:13 


28/07/00 12:22 office 182432 reports for CAP17 a receipts total of 
£412224.58 and a payments total of £430724.58. the difference is 
£18500.00. this office earlier raised a query because a transfer for 
an amount of £9250.00 seemed to have gone missing. the amount 


of the transfer is exactly half the amount of the difference between 
the receipts & payments. a call raised earlier PC0050974 was 


closed in error. 


28/07/00 12:27 GB082158 

Information: passing to EDSC as requested.” 
(emphasis added) 

A further entry is: 


“PM advises that the problem started when she transferred from 
counter No8 (by mistake - should have been from counter No1) 
£5590 to counter No3 and £3660 to counter No4 - while No8 was in 
the process of rolling over! Consequently, the cash left counter No8 
in CAP 18 and arrived in counters Nos3 and 4 in CAP 17. She 
eventually rolled the office, accepting the £9250 discrepancy and 
then transferred £9250 from counter No1 to counter No8. How was 
she able to transfer out of counter No8 while it was rolling over?” 


Part of the Post Office submissions are: 


“19. The net difference of these receipts and payments mismatches 
amounted to zero, meaning there was no overall impact on the 
branch. This would have been clear to Mr Coyne as the very nature 
of this issue is not that it causes money to be lost, but money to be 
accounted for in the incorrect time period. The PEAK shows that 
there are offsetting receipts and payments gains and losses and 
therefore it is possible to ascertain from the Peak whether the 
branch suffered a discrepancy. 


20. The issue was fixed through a planned software roll out that 
changed the code in the area that caused the bug (release C145) 


and the call was closed on 30 November 2000.” 

(emphasis added) 

That applies to the second PEAK but not to the one headed 
“Simultaneous Logon” which is PEAK PC0027581, which was 
closed in February 2002. 
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I find that this was a bug in Horizon which plainly had the potential 
to cause impact to branch accounts. The fact that in this specific 
case the discrepancy was later corrected does not, in my judgment, 
alter that characterisation. Indeed, on the Post Office’s own case, 
the accounts were affected (albeit corrected). Further, the way that 
PEAK PC0027581 unfolded - a lengthy PEAK which runs from July 
1999 to February 2002 - shows the unsatisfactory way some of 
these incidents were dealt with at Fujitsu. The entry of 9 November 
1999 states “....at first glance it could have a fair bit of business 
impact”. Even as late at August 2001 and January 2002 this PEAK 
was still given the code “Incident under Investigation”. One 
wonders quite how long Fujitsu would need properly to investigate 
something identified as “a bug in the Supplier’s Code” in July 2001. 
The final entry contains the sentence “Mr Lui [who is the SPM] is 
no longer employed by the Post Office and has not been for some 
years.” 


19. Post & Go/TA discrepancies in POLSAP. 
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This is during Horizon Online and occurred in 2012. It is accepted 
as a bug by the Post Office in paragraph 6 of Appendix 2, but is 
said to have had “transient” impact. The entry by Mr Coyne in the 
Bug Table relates to Horizon Issue 4 rather than 1. “Post & Go” are 
self-service terminals, that are no longer available in branch Post 
Offices, and are only available in Crown and WH Smith main 
branches. They are basically self-service kiosks, designed to avoid 
queuing. A customer can weigh a letter or parcel, the terminal will 
print the relevant stamps/labels, the customer deals with that 
themselves, and the item is then posted without the need for 
queuing. The Post Office submitted that this issue was irrelevant to 
the Horizon Issues trial as “this issue does not relate to branches 
that are the focus of this trial”. 


I do not consider that this is irrelevant as the Horizon Issues 
require consideration of bugs, errors and/or defects in the Horizon 
System. 


One of the PEAKs shows that Post and Go transactions were being 
accurately recorded by the relevant terminal and then transferred 
to Horizon. Horizon creates a TA (or transaction acknowledgement) 
of all the transactions on the P&G machine and, when accepted by 
the branch, the value of these transactions are added to the branch 
accounts. The data is also passed from Horizon to various other 
systems, including POLSAP. This incident was caused due to an 
issue at the reconciliation step within POLSAP. Fujitsu transfers 
the relevant data to POLSAP via BLE files. When POLSAP was 
comparing the Horizon data to the Wincor data, discrepancies were 
found as data for two (of the total of six) terminals in the branch 
was not being sent to POLSAP. The Post Office submissions state 
“There were six P&G terminals at the branch. Data from four 
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terminals only were being transferred to POLSAP because the two 
other terminals were not associated with conducting P&G 
transactions”. 


This is not an accurate summary of the PEAK. This states: 


“An example the customer has provided shows amounts of 115.05, 
46.88, 52.13 & 75.23 totalling 289.29 received on the file from 
Wincor and into POLSAP via BLE. 


The same (contra) amounts are also showing as being received 
from the branch when the TA has been accepted and are closed 
items in the account (netted off to 0.00). 


However, there is another amount of 289.29 which just has the 
date in the assignment field.” 


The Fujitsu response is: 


“Postings on the TfS call refer to a similar previous incident 
(A1040049 => Peak PC021943293), which was resolved between 
POL and Wincor Nixdorf; no details of this resolution are available 
to us. This incident is a week old, but only came to SSC late last 
night... The trading-date in this call, 2012-08-09, is three weeks ago 
which too old for us to be able to see the incoming file from Wincor 
Nixdorf... There is no evidence of a fault in HNG-X, and without the 
incoming file from Wincor Nixdorf there is nothing further for us to 
investigate. 


We can only suggest that POL do the same as they did with 
A1040049, and refer the matter to Wincor Nixdorf.” 


Anne Chambers records that: 


“Branch 020511 has many entries in the Subfiles_on hold report. 
This report should be monitored (by ?) to make sure problems are 
followed up - this should be resolved before closing this call. 


Horizon is receiving PG data for 6 separate PG tills at the branch, 
but only 4 of them have associated stock units. This causes the 
entire subfile for the branch to be Held, and the transaction data is 
not being sent to POLSAP. However the TA data for the 4 tills 
which are properly associated IS being sent through, and I think 
this is probably the cause of the POLSAP anomalies. 


The two unassociated tills are not doing any cash transactions - this 
is a known problem (see PC021870294), and means the PM isn't 
prompted to create an association. This may need fixing via MSC.” 


It was not that two of the six terminals were not doing Post & Go 
transactions, which is how the Post Office submissions seek to have 
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the PEAK interpreted. It was that they were not going cash 
transactions. There are types of transaction associated with Post & 
Go that are not cash transactions. They plainly are all six of them 
Post & Go terminals, as the PEAK records. If they were not doing 
Post & Go transactions at all, there would be no problem because 
there would be no subfiles on hold, or held. 


As Mr Coyne said in his supplementary report “a bug fix to the 
Horizon system was identified by Fujitsu, scheduled for 
implementation 13 September 2012 after 1800hrs and the 
Branches stock was to be corrected at 1700hrs that same day.” On 
17 September 2012 Anne Chambers herself reported in the PEAK 
that: 


“Following a change made centrally to facilitate this, the stock unit 
associations for the two new Post and Go terminals have been 
created by the branch and all the held external data (43 different 
days) has now been processed and passed through to POLSAP... 
We strongly recommend that POL monitor the SubfilesOnHold 
report which is sent to them daily, so that any other external 
terminals with problems can be investigated quickly in case a 
similar correction is needed”. 


The “change made centrally” can only have been to the software. It 
is also the case that the underling bug, error or defect was having 
an effect for 43 days, which is in excess of one trading period which 
are four weeks long (and occasionally five weeks) but both those 
periods are shorter than 43 days. It is also the case that POLSAP 
was wrong for that period, which is a factor which impacts upon 
the dispute between the parties about the use of ARQ 
data/management data. The Post Office submits that it was “quickly 
resolved” but I reject that. 43 days, or over 7 weeks, is not 
resolving this bug (or its effects) quickly. 


I find that this is a bug with potential to cause impact to branch 
accounts. I do not know the date when the Post & Go service was 
withdrawn from branch post offices, or how many branches had it 
during the period when it was so available. The statements made in 
respect of this in the Post Office submissions are said, in the 
clarification table, to be based on “instructions from [the] Post 
Office”. The information is vague. I accept that this service is only 
now available in WH Smith and Crown branches, but other details 
cannot be specified because there are none available from any 
source. 


20. Recovery Failures. 
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The entry in the Bug Table identifies this as relating to Horizon 
Issue 4. These arise under Horizon Online and not Legacy Horizon. 
The Post Office does not accept these as a bug and states that there 
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is no evidence of a bug in Horizon; it refers to the subject matter of 
the different PEAKs as “issues”. There are three different PEAKs 
relevant to this. Mr Coyne accepted in his cross-examination that 
this (or these) should be removed from the table that he used as 
the originator for his list of bugs (which was Mr Coyne’s Table 1). 
Dr Worden stated in the bug table that “there was some implication 
of hardware faults, with a replacement of a base unit, but the PEAK 
has no evidence of software faults in Horizon.” In one of the PEAKs, 
he concluded that there was no evidence of any fault in Horizon. 


The situation in terms of PEAK PC0220532 is unsatisfactory. The 
Post Office submit that this “relates to a cash discrepancy that was 
only raised by the Subpostmaster with Post Office and Fujitsu nine 
months after the issue occurred.” 


It is therefore clearly submitted that this took nine months for the 
SPM to report at all. This is based on the date of the PEAK (15 
September 2012) and the date given in the very first entry that this 
happened on 6 January of the same year. However, that is simply 
incorrect on the face of the document. The date of the PEAK does 
not mean that the SPM took that period to report the matter to the 
Helpline or their manager, for example. Reporting in January by 
the SPM is in fact recorded in the PEAK itself, but not in the first 
entry. Further, that submission ignores these entries later in the 
PEAK for 6 September 2012 that refer to a TFS call being raised on 
7 January 2012: 


“However I have checked our call logging system which has 
provided some useful information. According to our archived TFS 
call login system, the Node: 5 did have ?Physical memory Dump? 
on 6/1/12 and a TFS call (5043049) was raised. The node was 
swapped on 7/1/12.” 


On 13/1/12, PM raised another TFS call (5060273). PM reported 
that she had a loss of £300, she wanted to know if this was related 
to the Base unit fault. PM was referred to NBSC to investigate the 
discrepancy. 


On 13/1/12 NBSC raised a TFS call (5060420) and asked for the 
discrepancy to be investigated. 


On 13/1/12 @16:32pm, the frontline helpdesk contacted the PM 
and the following information was logged on the call. 


?Spoke to the PM who states that she had the loss before the base 
unit was swapped out the loss was for 190.00 PM states that she 
rolled over her TP on the Wednesday which had no loss or gains. 
PM states that she then did a transaction log on the 5th Jan this 
shows a loss of 190.00 The base unit got replaced on Saturday 7th 
Jan. 
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PM states that she had a different error message on the system 
every day but did not record the error message PM states that she 
had physical memory dump message come up on the Friday 6th Jan 
and the base unit was then swapped on the 7th Jan?” 

(emphasis added) 


Later entries in the PEAK state: 


“Assuming the cash declaration on 04/01/12 was correct, NBSC will 
need to go through the transactions that were done in BB during 
this period (possibly by use of Transaction Log), to try to determine 
where the discrepancy came from. The fact that on 04/01/12 there 
was a negative cash discrepancy balanced out by a similar but 
positive stamps discrepancy, may be related to this issue. ? 


It is not clear what investigation/s were carried out by NBSC and 
no further TFS call was raised with regarding to this discrepancy. 


If further investigation by Fujitsu is required, Post Office will have 
to request that the branch transaction data is retrieved from the 
audit server. If there is any possibility that this is required for 
litigation, it must come through the Security (ARQ) route. 
Otherwise queries of this nature should be sent via Mark Wardle at 
POL, and should be routed to the reconciliation team in the first 
instance. Such requests may be chargeable.” 


This trial is not to determine individual discrepancies, although 
they are identified where relevant to the Horizon Issues. On the 
information before the court on this item, it is not possible to 
resolve this. This means that there is insufficient evidence to make 
a finding of a specific bug in this case, or on this PEAK. 


The second issue under this heading related to PEAK PC0241242 
on 23 February 2015. This caused a discrepancy and the Post Office 
submits that “a Transaction Correction was done for the shortage”. 
The shortage was £70.66, and the claimants challenge the Post 
Office on the origin of this positive statement that this was done. In 
clarification, the Post Office relied upon Fujitsu stating in the PEAK 
that the Post Office needed to do this. The PEAK states on 5 March 
2012 at 1546 hrs: 


“This office was doing the following cash withdrawal txn for £70.66 
on 23/2/15 @10:46. 


The session also contained a non financial Health Lottery txn. 


The settlement failed due to poor communication with the data 
centre. The disconnected session receipts were printed which 
advised PM to pay the money out. Spoke to PM who confirmed that 
the money exchanged hands. 
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When PM tried to log back the recovery kicked-in. However the 
Health Lottery APADC recovery script failed and left counter 
unusable. 


Yesterday POL Branch Support team authorised wus to 
remove/update the session. 


Today I have carried out and completed the task. PM is now able to 
use the node again. 


Reconciliation needed for the banking transaction: 


The cash withdrawal txn was authorised and PM said they paid the 
money out. 


This will leave this office £70.66 short (cash shortage) as the 
session not completed fully. POL need to do appropriate 
reconciliation; transaction correction.” 


However, a later correction in the PEAK for 5 March 2015 at 16.06 
states “MSU please note that it was a CAPO cash withdrawal for 
£140 and NOT £70.66.” This is repeated at 16.12. The entry for 9 
March 2015 states “I have also advised MSU to do the necessary 
reconciliation and raise BIMS with POL for the Cash Withdrawal 
txn for £140.” 


It is clear therefore that the root cause of the issue was the failure 
of the AP-ADC recovery script (Automated Payment Advanced Data 
Capture script, which is written by Atos). This therefore directly 
shows that there was an error or defect in Horizon, and this led to 
a potential impact upon branch accounts. 


For the purposes of the Horizon Issues trial, this is relevant to 
Horizon Issue 4, as explained by Mr Coyne. 


The third issue shown in another PEAK is PC0197643 dated 14 
April 2010. This is a failure of automatic recovery. This will 
inevitably happen in any system, as accepted by Mr Coyne. 


The Post Office submitted in respect of all three of these PEAKs the 
following: 


“For the reasons set out above, this is not a bug at all. There are 
three distinct issues under this heading. There is no evidence of a 
bug in Horizon and in any event, instances of any issue were 
caught by automatic reporting.” 


That is correct as far as it goes - the PEAKs are not evidence of 
bugs. The first two are however evidence of errors and defects in 
Horizon. They do however go to Horizon Issue 4, not Horizon Issue 
1. 


21. Transaction Correction Issues. 
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This arises under Legacy Horizon. This is de facto accepted as a 
bug by the Post Office, but it is submitted it had no branch impact. 
Mr Coyne’s entry in the Bug Table was that “Transaction 
Correction bugs/errors and defects do not cause discrepancies with 
branch accounts but” and he then listed various consequences. In 
his report he referred to “technical flaws”. He also in his cross- 
examination accepted that he did not consider this to be a bug with 
lasting financial impact. Dr Worden, in respect of some PEAKs 
under this heading, accepted they were evidence of a bug, such as 
PEAK PC0129587 where he stated “In my opinion this bug would 
result in an inconvenience to the Subpostmaster (inability to 
rollover to the next TP) but would not result in inaccurate 
processing of any TC, or any impact on branch accounts” (emphasis 
added). The bug is included by the Post Office in the list of 
paragraph 5 of Appendix 2 under the heading “the following bugs 
had no branch impact.” 


The submissions by the Post Office is that there is “no evidence of 
any financial impact upon branch accounts”. It is therefore 
effectively accepted that it is, or there were, bugs responsible for 
these issues, but it is also effectively agreed by the parties that 
there would have been no financial impact upon branch accounts. 


Transaction Corrections were introduced in September 2005, 
replacing the previous method which were called Error Notices. 


Some of the PEAKs, such as PC0120459 dated 4 May 2005 and 
others from around that time, refer to issues with the TC button on 
the screen at that terminal in a branch not working as it should. 
The so-called “pick list” shown on the screen would freeze. These 
issues were logged by the team at Fujitsu called the Solution 
Validation and Integration team. This was part of testing. None of 
the documents referred to by the Post Office in its submissions to 
justify the submission “As such, these PEAKs all relate to issues 
raised by users during testing prior to the TC functionality going 
live in September 2005 (when it replaced error notices)” were 
referred to in the trial. I do however accept that they show that the 
TC process was tested prior to release to the whole branch 
network, and other PEAKs also show this. The Post Office also 
submits that this was not “a bug at all”. If that is right, then it must 
have been an error or defect, but given that the documents show 
that it was fixed during the testing process, the precise terminology 
probably does not matter. 


Other PEAKs however relate to the period after September 2005 
when the testing process was over. PEAKs PC0129587 and Peak 
PC0130056 refer to the Petersfield branch, and are dated 4 
December 2005 and 14 December 2005. They both relate to the 


same issue and the latter is what is called a clone of the former. 
The first PEAK relates to a TC for a £9,000 debit and part of that 
PEAK states: 


“01/12/05 16:01 Caller states that each time they try to do a 
transaction correction the counter is freezing up and going to 
system busy, has attempted on 3 different counters and the G/W 
was on system busy for about 2 hours, pm has had to reboot 3 
times attempting to do this. 


01/12/05 16:07 uk955556 


Advice: Asked pm to provide details regarding transaction and 
fault. 


01/12/05 16:08 uk955556 


Information: pm has attempted to run this correction on 3 counters 
now, 1, 5 and 2 and each time the counter is freezing up and going 
to system busy, will not come back from this. 


01/12/05 16:09 uk955556 


Information: Transaction was for a premium bond on the AP 
system. 


PM was working on stock unit - E at the time. 

01/12/05 16:10 uk955556 

Information: Transaction was carried out on 29th October at 12.23. 
01/12/05 16:10 uk955556 


Advice: Advised pm that i will have to look into this and get back to 
her. 


01/12/05 16:10 uk955556 


Information: pm is fine with this but states if it is not resolved by 
14th December she will be unable to rollover. 


01/12/05 16:32 uk955556 


KEL Ref No.: checked for kel's but all kel's found on transaction 
corrections refer to specefic events which are not present in this 
case.” 


The lack of previous KELs on this subject at that point means that 
this was a new issue which had not arisen before. It cannot 
therefore be put down to testing which was completed by then. 
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A later entry on 6 December 2005 is: 
“Several interesting points to note: 


(a) 111111111 subscription group on core server contains 35 more 
msgs than on the counters: 3280 compared with 3315 msgs. The 
extra msgs are just Bureau: CurrencyRates and CSvcMargins msgs. 


(b) A couple of times after getting tired of waiting for the TC to 
process, the Clerk did a session transfer to another counter which, 
according to EP/SPG/001, should not be possible. 


(c) I have reproduced the problem by importing the mstore onto 
the SSC counters and setting a cut off date of 01/12/05 11:27:10 
which was before the PM?s first attempt. 


(d) Peak PC0120459, raised on S80 E2E XI, reported the same 


symptoms and the root cause was found to be missing/incorrect ref 
data. This call was investigated by Mike Coon.” 


(emphasis added) (question mark present in original as “PM?s”) 


“Ref data” means reference data, which is part of the software of 
Horizon. In my judgment this shows clearly that this was a bug. 
This was therefore known about by Fujitsu in 2005. 


I shall quote some paragraphs of the Post Office’s detailed 
submissions on the next PEAK: 


“10. Peak PC0129587 relates to a report by a SPM on 1 December 
2005 that each time she went to select a TC for a £9,000 debit, 
which arose from an incorrect entry for Premium Bonds, her screen 


froze and she was unable to accept it. 


11. Peak PC0129774 and Peak PC0130057 refer to the Bosham 
branch. These PEAKs relate to a report by a SPM on 6 December 
2005 that each time she went to select a TC, a £22,500 debit for an 
incorrect entry for Premium Bonds, her screen froze and she was 
unable to accept it. 


12. Mr Coyne’s analysis proceeds on the basis that this issue was 
only experienced by these two branches. Mr Coyne fails to refer to 
KEL LKiang2837P [dated 5 January 2006] which demonstrates that: 
50 other branches reported an issue with their screen freezing to 
Fujitsu during December 2005; 48 branches reported that this 
prevented them selecting an outstanding Camelot Lottery TC and 
rolling over into the next trading period; and 4 branches reported 
that this prevented them selecting an outstanding Premium Bond 
Sale TC and rolling over into the next trading period, including the 
two branches referred to in the 4 PEAKs mentioned by Mr Coyne. 
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13. The diagnosis of the issue was that the text drafted into the TCs 
contained a string of 35 characters without a space. The code to 
put each TC on a screen was attempting to split the text at a space 
(as is normal for a word processor), but the code was unable to 
process a string of this length. 


14. The issue was resolved quickly. Having first been reported on 1 
December 2005, a work-around was developed whereby the 
affected branches were instructed to roll over all stock units except 
one (the one relating to the unprocessed TC). This enabled them to 
roll over into the new trading period. A software fix was then 
released on 22 December 2005 which enabled the branches to 
select and process the outstanding TC, and roll over the last stock 
unit. To avoid the issue reoccurring in future, Post Office 
confirmed that strings of this length without spaces would no 
longer be included in the text of TCs. 


15. Although included in Mr Coyne’s analysis of branch affecting 
bugs, the impact of Issue 2 [ie this issue in this PEAK] was limited 
to the affected branches being unable to roll over to the new 
trading period for a short period of time. It had no financial impact 
on branch accounts.” 

(emphasis added) 


The code explanation above was not contained in any evidence and 
was taken by the Post Office in its submissions from documents 
that were not even deployed in the trial. However, these show that 
it was clearly a bug, which Fujitsu knew about in late 2005 and 
which was fixed by means of a software fix at the very end of that 
year, with a KEL being produced in January 2006. TCs were 
introduced in September 2005. 


There was no impact on branch accounts, as accepted by Mr 
Coyne. This therefore goes to Horizon Issue 4 and not Horizon 
Issue 1. 


There is another PEAK from September 2010, PEAK PC0204350 
which refers to Horizon Online and a report by an SPM on 14 
September 2010 that he had suffered an £80 cash loss which he 
believed was due to a “system error’. The PEAK itself states at the 
beginning “Summary: can you please look into this problem. Our 
second line cannot find any user error.” The SPM referred the 
Fujitsu representative on the call to a TC report he had generated 
on 10 September 2010, which had allowed him to request a date 
range of 60 days, but which did not provide him with data on TCs 
which were more than 40 days old. 


The PEAK indicates that on 14 September 2010, Fujitsu requested 
detail of the specific transactions relating to the £80 loss from the 
SPM in order that it could carry out an investigation into the issue. 


No information was provided by the SPM. Given the absence of 
evidence, the matter was thereafter closed by Fujitsu as Category 
95 “Final - Advice after investigation” and “no fault in product”. I 
find that this is a misleading description ascribed to close this 
PEAK. It also states “TC data is kept on the live database [BRDB] 
for 40 days so any txn query for a TC where it was put through the 
system more than 40 days previous WILL NOT show up in the 
branch. System is working as designed.” The first sentence of this 
is correct. Those timescales are indeed what occurred. However, 
the entry “system is working as designed” is at odds with the fact 
that second line support at Fujitsu had already said there was no 
user error. The penultimate entry in the PEAK also states 
“however, beyond that, I will have to request archived data from 
our Audit Team in order to confirm those TC txns in July 2010.” 
There is no evidence to suggest this was done; the matter was 
simply parked without any proper resolution. I find this to be 
evidence of a bug. Given that Mr Coyne stated in cross-examination 
that there was no evidence of impact on branch accounts (although 
the £80 referred to might suggest otherwise) I take this into 
account in relation to Horizon Issue 4. However, it plainly has the 
potential to cause impact to branch accounts and therefore should 
also be taken into account under Horizon Issue 1. 


22. Bugs/errors/defects introduced by previously applied PEAK fixes. 
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This also arises in Legacy Horizon. It is de facto accepted as a bug 
by the Post Office, but it is submitted that it had no branch impact. 
It is included in paragraph 5 of Appendix 2 under the heading “the 
following bugs had no branch impact”. It is also submitted by the 
Post Office that “a number of PEAKs arose in testing. The PEAKs 
that arose in the live environment do not indicate evidence of 
branch impact.” It is listed in the Bug Table as concerning Horizon 
Issue 4. Mr Coyne stated in the Joint Statement that “Branch 
accounts would be affected by this bug which would cause a 
discrepancy when handling cheques where the value of the cheque 
would be doubled” although he accepted that the SPM in question 
was processing a cheque in a different manner to that 
recommended. Dr Worden used the term “a fault” in his entries, 
maintained that there was no impact on branch accounts if the 
SPM followed correct procedures, and he also (in respect of 
another branch) relied upon the fact that the effect was an error of 
2p only. 


The Post Office submits that there are three distinct issues and that 
they were all grouped under the same head. Some of the PEAKs are 
from 2000 and one is from 2004. 


PEAK PC0053160 refers to a report by a Fujitsu test team member 
on 29 August 2000 which concerned an issue with the Training 
Counter which froze when completing a transaction log report. 
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Further testing by Fujitsu demonstrated that the issue could also 
be present in the live environment, and a software fix was deployed 
on 6 September 2000. Mr Coyne concluded that the fix 
implemented what are called regression bugs. There are references 
to this in the PEAK. A regression bug is something that prevents a 
function from operating as it should. Entries in the PEAK such as: 
“Training Counter freezes using transaction log 

Training Counter freezes when miskeying using transaction log. If 
you do the following: 

perform sales transaction 

select Reports/Transaction Log 

select Value from .10p, Receipt 

select value to £100, Receipt 

Continue (F16) 

Complete (F16 from Print screen, instead of F4) 

Reprint (F4 from Reports menu) 

you go back to the Transactions Log menu as expected but the 
counter freezes and you can't get out. Continue and all the other 
buttons do not work (except to flash when selected). 

This manifests itself in training, if a delegate mis-hears or miskeys 
a keying sequence in doing a transaction log report. Could be a live 
issue too.” 

And: 

“This is NOT a Traing Mode issue, but an EPOSS issue. The 
problem is that if you Prev out of the Report/Reprints menu you are 
returned to a previous 

Transaction Log selection menu from which there is no escape. 
Dean has not specified which build he was using, but I have 
reproduced this 

on both CI4L1 and CI4R streams.” 

And 

“F} Response: 

The problem described at the start is fixed. However, application of 
the work packages to the training counter appears to have caused 
some regression, namely: 

If you perform a number of different types of transactions (e.g. 
passports, APS, BT bills) the daily reports are correctly populated, 
however no reports are listed when you press the Summaries icon 
(mandatory or otherwise), and the system does not require you to 
print and cut off the mandatory reports before rolling over the 
stock unit into the next CAP. 

[END OF REFERENCE 21808091] 

Responded to call type P as Category 42 -Product Error Diagnosed 
The response was delivered on the system” 

(emphasis added) 


These suggest that Mr Coyne is right - Fujitsu itself diagnosed it 
both as a regression issue and also diagnosed a product error. The 
Post Office’s written submissions stated that “further investigations 
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by Fujitsu have concluded that rather than a regression, this was 
an error in the way that the test rig was setup; an approved 
combination of work packages was not being used” although this 
was Challenged by the claimants as not being contained in any of 
the evidence. 


The answer to the challenge that was given was that this was taken 
from a document not deployed in the trial, and from common sense. 
The document was PEAK PC0053160. The entry expanding on 
common sense states “the developer is stating that the tester failed 
to re-test against an approved combination of work packages; 
hence the assertion by the tester that there was a code regression 
is incorrect”. It is also stated “work packages sometimes have 
interdependencies in order to provide a complete fix to any given 
issue.” I consider that the PEAK document was deployed in the trial 
in the sense that it is referenced by Dr Worden in his entry in the 
Bug Table, but in any event the entry relied upon is 22 September 
2000 when Les Ong stated: 


“It's not stated which build this is but, by adding two M1 work 
packages in isolation, I'm not surprised there's further problems. 
These two WPs contain EPOSSCore, EPOSSReportbroker, 
EPOSSStockunit and BESReports. Any of these may have 
dependencies on other WPs or Ref Data. 


You need to stick with a CI4L1, CI4R or current M1 build. At least 
bugs in these are known and can be quoted. 


[END OF REFERENCE 21817364] 
Responded to call type P as Category 74 -Fixed at Future release” 


This is clearly a reference to a bug, and is also something that 
arose on a training rig, but it is in my judgment a regression issue. 
The tester thought that it was, and that would have been one of the 
things that the tester was looking out for. 


In any event, the next PEAK clearly was a regression issue. I will 
quote the Post Office’s submissions: 


“8. Peak PC0098230 refers to an issue reported by a SPM on 13 
January 2004 relating to a discrepancy with his cash account. 
Fujitsu called the SPM to obtain further details and ascertained 
that when the SPM was declaring his cheques, the value of cheques 
declared as stock doubled. 


9. The issue was diagnosed as a code regression relating to the fix 
implemented in Peak PC0097081. A work-around was proposed two 
days later to the SPM, on 15 January 2004, that the SPM should 
follow a different procedure when declaring his cheques. A 
software fix was thereafter released for the code regression.” 
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The PEAKs relating to the third issue were regression bugs but 
were discovered during testing in early 2000. 


Therefore, in my judgment, Mr Coyne is correct in that the PEAKs 
that were the subject of this heading in the Bug Table, were 
regression bugs. Given the lack of impact upon branch accounts, 
these relate to Horizon Issue 4. The one in 2004 is particularly 
notable, but given it was resolved so very quickly (a matter of days) 
there was in fact no impact on branch accounts, but there was 
potential impact only. I consider this bug to be relevant to Horizon 
Issue 4. 


23. Bureau de change. 
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These occurred in both Legacy Horizon and Horizon Online and 
arose in 2005, 2006 and 2010, and should be differentiated from 
bug 14, which arose in 2017, which was during Horizon Online and 
is referred to as Bureau Discrepancies. The Post Office does not 
accept this as a bug. The Post Office also states that Mr Coyne has 
identified three issues all with the same heading, and submits that 
they all relate to user error. Dr Worden’s entry in the Bug Table in 
relation to one of them states “Analysing the second KEL (2010) I 
noted: ‘Impact small until bug fixed - rounding errors 10° in 
exchange rates.’ The Post Office’s submissions that this is not a bug 
is a somewhat bold submission, given the KEL to which Dr Worden 
refers expressly states, in terms, that the problem is a different 
exchange rate appearing on the HNG-X rateboard and the Horizon 
rateboard, two different parts of the same system. It also ignores 
Dr Worden’s statement “until bug fixed” (emphasis added). HNG-X 
is Horizon Online, but this cannot applies to the occurrences in 
2005 and 2006. 


KEL AgnihotriV917N dated 23 June 2010 states within the KEL 
itself: 


“Problem - there is a bug in the code” (emphasis added). 
The full entry reads as follows: 
“Problem 
There is a bug in the code. 
Steps to reproduce:- 


1. In spotrates.xml make the value for any currency (say Egypt 
Pound EGP) as 10 and in margins.xml as 1.004 under BUY- EGP. 


2. Start the Counter 


3. Configure RatesBoard with this currency. 
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A. Click on update. 


5. Under Buy Notes column the value displayed for EGP will be 
10.1 instead of 10.100. 


Solution - ATOS 


This bug is harmless as the business rule is working as desired 
except for displaying effective rate. 


It means in the background the value used for effective rate will be 
as calculated from the formula but when it comes to display, value 
will be stripped of trailing zeroes. 


See also PC0200042 and KEL Agnihotriv245L. 


Solution in progress.” 
(emphasis added) 


There plainly is a bug - it is referred to twice in terms in the same 
KEL. The author of the KEL states that it “is harmless”. The other 
KEL referred to, KEL Agnihotriv245L, is by the same person. It 
refers to the fact that customers will (or may) be given the wrong 
rate to the one displayed in the branch. It states: 


“There is a bug in the code. 
Steps to reproduce:- 


1. In spotrates.xml make the value for any currency (say Egypt 
Pound EGP) as 8.3325 and in margins.xml as 11.2400 under BUY- 
EGP. 


2. Start the Counter 
3. Configure RatesBoard with this currency. 
4. Click on update. 


5. Under Buy Notes column the value displays for EGP will be 
9.269 instead of 9.2691. 


Although the system does use the correct values to calculate the 
rates and the issue here is with the display only. However, because 
the display is on the Customer facing rates board there is potential 
for annoying Customers. They may get a slightly different rate to 
that advertised on the board. And may challenge Post Masters on 
this issue. 


As the issue relates to the 5th or 6th significant figure the impact 
will be fairly low - and in favour of the Customer 50% of the time 
e.g. if I buy £1000 of XYZ currency at a rate of 99.12 then I expect 
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to get 99120.00 in XYZ (according to the board) but I will actually 
get 99115.90 in XYZ (by the Counter) - a difference of 4.10 XYZ or 
the equivalent of £0.04. Its effect may be magnified by high value 
transactions but generally, different rate bands apply in these cases 
- so the rate board display is inconsequential.” 


Solution - ATOS 


The solution is a code fix to change the code to treat values in the 
affected ranges correctly. 


See also PC0200090 and KEL AgnihotriV917N 
Solution in progress.” 


This plainly shows a complacent, if not lackadaisical, attitude to 
financial precision. The error may be in the customer’s favour 12 
the time, but that means it will be in the Post Office’s favour the 
other 1⁄2 the time, and the customers will not be the same person on 
both occasions, and are highly likely to be entirely different people. 
The KEL also recognises that “its effect may be magnified by high 
value transactions”. 


The solution in one of the other KELs, raised by Anne Chambers 
namely AChambers2252R is: 


“Solution - ATOS 


If a PM reports a loss connected with a currency transaction that 
was reversed, it is possible that the reversal had not been carried 
out successfully.<br><br>Ask the PM to check the Reversal 
Receipt. If this shows<br><br>Cash FROM CUSTOMER 
750.00<br>Cash TO CUSTOMER 750.00<br><br>they have 
reversed just the cash settlement part of the transaction, which has 
no overall effect. The currency and margin part of the transaction 
has not been reversed.<br><br>Do a transaction log search using 
the Session Id from the original receipt, or by 
date/time.<br><br>This should show 3 elements - for 
example<br> <br>2-29826-2 SC Euro 1- 720.00-<br>2-29826-2 
SC Curr Sell Margin 1- 30.00-<br>2-29826-3 SC Cash 1 
750.00<br><br>The element which should be reversed is 2- 
29826-2 (i.e. Euros and margin). As long as the PM has not yet 
rolled over the stock unit, they should be able to reverse this 
transaction now.<br> <br>If the stock unit has been rolled over, 
NBSC will have to advise on what can be done to resolve the 
system loss relating to this transaction.” 


They are therefore two different issues. The Post Office, again 
boldly, submitted at paragraph 2 on page 511 (which is part of its 
Appendix 2) in relation to this bug that “There is no evidence of a 
bug in Horizon. Each issue is an example of user error in Horizon.” 
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I find those submissions to be verging on the extraordinary. The 
substantive judgment to which this is an appendix identifies what 
the Known Error Log is. The entries above make it clear that there 
is a bug - the very word chosen by the Fujitsu employee who wrote 
the two KELs is “bug”. To see this characterised in submission as 
there not being a bug, and being evidence of human error, is not 
only puzzling but flies in the face of the terms in the Fujitsu 
documents. I find that it is evidence of a bug. One of the PEAKs 
only of those listed in the Bug Table, namely PC0137437 is user 
error, a point identified by both Mr Coyne and Dr Worden. 
However, simply because one PEAK is this, does not mean that the 
summary submissions can be accurately stated as “There is no 
evidence of a bug in Horizon. Each issue is an example of user 
error in Horizon.” That is simply wrong in fact. 


Further, Mr Coyne identified in his evidence that there were 8 
PEAKs associated with these KELs that fall in the date range of 
2010 to 2018. 


There are some unsatisfactory aspects to the way the Post Office 
tries to explain its position on this bug. At paragraph 14 of its 
submissions, it submitted that the Post Office subsequently 
“reviewed ARQ data for the branch” in question on one of the 
PEAKs, and that “there is nothing to suggest that the branch’s 
remming in/out of foreign currency caused a loss in branch.” When 
challenged by the claimants that this was not in the evidence, the 
Post Office clarified by stating this came “on instructions from Post 
Office, which did not have an opportunity to adduce evidence on 
the point”. That is not correct. This submission was made in 
relation to PEAK PC0151787 which is the second entry in the final 
column in the Bug Table itself, headed “supporting evidence”. It is 
also referred to, correctly, in paragraph 4 of the Post Office’s 
submissions in the following terms “In JS2 Mr Coyne mentions 
three PEAKs under the heading of “Bureau de Change””. JS2 
means the 2™ Experts’ Joint Statement, which was dated 25 
February 2019, well before the start of the trial. There was such an 
opportunity, as supplementary questions in chief could have been 
put (or permission to do so asked) and it is wrong to suggest that 
the Post Office was denied the opportunity to meet this evidence. 
Dr Worden chose to adopt a different way of giving his expert 
evidence, namely very high level. 


In any event, there is nothing in my judgment to justify a finding of 
user error which is relied upon by the Post Office in relation to this 
PEAK. The first entry in the PEAK states, not user error, but the 
following: “PM states that that he has remmed out some foreign 
currencies, reversed them and re-remmed them but when he has 
come to do the stock balancing his main stock unit is £907.97. 
Richard at NBSC 2nd line has requested that I log a call and send 
to SMC.” Even Fujitsu categorise the call closure as “Defect cause 
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updated to 42 -- Gen - Outside Pathway Control.” That is not 
consistent with user error. 


In any event, I find that there is such a bug and I take this into 
account in my overall conclusions on Horizon Issue 1 as well as 
Horizon Issue 4. 


24, Wrong branch customer change displayed. 
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This is a Legacy Horizon Issue and occurred in 2005. Mr Coyne 
concluded this was a bug and part of his entry in the 2™ Joint 
Statement was “the KEL explains that ‘the cash amount entered is 
multiplied by the Qty and hence the new stack total is wrong’”, and 
that this Horizon bug was due to incorrect reference data. The role 
that reference data plays in the operation of Horizon has been 
explained. Here, this led to an incorrect amount of change being 
displayed on the branch screen leading to the operator (which 
means SPM or their assistant) providing the branch customer with 
the wrong amount of money, thereby leaving a discrepancy in 
Branch Accounts. He also stated that “It is possible that the amount 
of change shown on screen is more than the actual money tendered 
by the customer.” Dr Worden’s entry was “When analysing this 
KEL I noted ‘Sounds like a genuine problem which may have led to 
giving the customer the wrong amount - i.e. not recoverable.’ It is 
now accepted by the Post Office in paragraph 7 of Appendix 2 as 
one of a number of “bugs with lasting impact (although they were 
resolved)”. It is also said that it is a reference data bug, and that 
therefore once discovered could be quickly fixed by changing the 
relevant reference data. 


The summary of the Post Office’s submissions are as follows: 


“2. Bug 24: Wrong Branch Customer Change is a bug with the 
potential for lasting financial impact. This is a reference data bug. 
The experts have agreed that while reference data bugs may be a 
significant proportion of the bugs with financial impact, once 
discovered, they could be quickly fixed (by a change to the 
reference data) once the bug is correctly identified. This was the 
case with Bug 24. The issue would have visible to the SPM as the 
incorrect quantity would have displayed on the screen. Fujitsu 
identified the root cause and developed a fix within two weeks of 
the issue being reported by the SPM. 


3. This issue relates to quantities of stamps and postage labels 
(Smartpost Transactions) not correctly resetting to 1. 


4, When a quantity of greater than 1 was entered for a Smartpost 
Transaction, the quantity was not reset to 1 when the clerk moved 
on to the settlement screen. This could result in subsequent items 
in the session being multiplied by whatever quantity remained and 
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could affect further items being sold or the amount being tendered 
towards settlement. 


5. Peak PC0128264 was opened on 4 November 2005 as a result of 
a SPM reporting the issue on 4 November 2005. The matter was 
passed to Fujitsu’s development team for a software fix on around 
10 November 2005 and a fix was implemented on 18 November 
2005. The Peak shows that Fujitsu suspected that the problem was 
introduced by changes to the Smartpost Transactions that had 
been implemented from 24 October 2005. 


6. On 6 December 2005, a further instance was reported (Peak 
PC0129791). The root cause was identified and it was found that 
this issue related to Peak PC0128264 which documented the fix 
that had been put in place. On 7 December 2005, Fujitsu found 
that the fix had not been applied to a group of branches and the 
reference change data fix was then implemented overnight to the 
remaining branches.” 


The KEL on this was again one authored by Anne Chambers. It is 
undoubtedly a bug. Not only was it a bug, but once it was 
discovered and a software fix introduced, it persisted because “the 
fix had not been applied to a group of branches”. No detail is given 
as to how large that “group of branches” was - the only description 
in the submissions is that it was “was an active group of branches, 
group 111111112”. Nor is there any explanation available in the 
evidence as to why the fix, which was known to be required to fix 
this bug, was not applied to every single branch, and why or how 
this “active group” of branches was omitted. 


I find that it is a bug with potential to impact upon branch accounts 
and I take its existence into account in my answers to both Horizon 
Issue 1 and Horizon Issue 4. 


25. Lyca top up. 
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This is a Horizon Online bug. Lyca is a type of mobile phone “top 
up” card which allows people to pre-pay for mobile phone services. 
This is accepted as a bug by the Post Office in paragraph 6 of 
Appendix 2, but is said to have had transient impact. It is also 
accepted as being a reference data bug, and paragraph 2 of the 
detailed part of the submissions on this in Appendix 2 states “Lyca 
Top Up is a bug with the potential for lasting financial impact. This is 
also [a] reference data bug. As set out above, the experts have agreed 
that while reference data bugs may be a significant proportion of the 
bugs with financial impact, once discovered, they could be quickly 
fixed (by a change to the reference data) once the bug is correctly 
identified.” It is also submitted by the Post Office that it was identified 
through Fujitsu’s automatic reporting. 
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The nature of the bug, which occurred in August 2010, is that a 
customer will pay the SPM a certain sum (for these purposes, an 
example of £20). The transaction is entered on the Horizon counter 
by the SPM and, once the transaction is processed by Horizon and 
authorised by E-Pay (the financial institution), a receipt is printed 
for the transaction which the SPM should provide to the customer. 
It is this receipt that contains the customer’s voucher to apply the 
top up to their mobile phone. Applying that voucher will give the 
customer £20 of credit to use on their phone. 


The problem was picked up by SPMs who reported it to the NSBC, 
and also by Fujitsu through its NB102 reporting. Depending upon 
what happened when the customer and the SPM did this 
transaction, would affect whether branch accounts were adversely 
affected. However, the Post Office accepts that “if the SPM 
recovered the transaction and incorrectly confirmed on Horizon 
that the top up had been successful, despite no top up receipt 
having been printed, the transaction would be recorded in the 
branch’s accounts, meaning the branch would likely have 
experienced a shortfall to the value of the top-up as no money 
would have been taken from the customer. If the SPM logged back 
into the counter and correctly confirmed that the transaction had 
not been successful, a zero value transaction would be recorded in 
the branch accounts and a reversal generated for the top up. This 
should have resulted in the affected branch accounts being correct, 
but due to the reference data issues the reversal being sent to E- 
Pay caused E-Pay to treat the top-up incorrectly as a successful 
transaction.” 


The PEAK shows that it was described as a “system error during 
transaction” and that it was resolved by means of a change to the 
RDT generator mechanism. One entry also states that “This 
problem could also show up as incorrect Welsh on receipts as 
special Welsh characters may also be incorrectly translated.” The 
corrective reference data was released to live to take effect 
overnight on 20 August 2010. 


I find that it was a bug and I take its existence into account in my 
answers to both Horizon Issue 1 and Horizon Issue 4. 


26. TPSC 250 Report. 
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This is a Legacy Horizon bug. It is accepted by the Post Office in 
paragraph 7 of Appendix 2 as one of a number of “bugs with lasting 
impact (although they were resolved)”. The origin of the experts 
discovering this bug is a KEL raised by Anne Chambers in February 
2005, and last updated in April 2008. 


There are approximately 24 different PEAKs dealing with this bug. 
Their date range is between 2005 and 2009, with the majority of 
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the incidents occurring in 2005. It relates to the printing of labels 
for postage. The KEL was originated by Anne Chambers. 


The Post Office has identified, within its submissions which deal 
with all of the PEAKs, a number of different sub-issues. The 
summary paragraph states: 


“1. The experts have drawn together five distinct issues under the 
heading TPSC 250 Report..... 


2. This was a backend reporting problem and so the chances of 
branch impact are small. Of the five issues: issue 1 resulted in 
incorrectly flagged exceptions but no reconciliation was required; 
issue 2 resulted in no financial or operational impact on a branch 
accounts and was limited solely to the process of copying/ 
harvesting transactions to Post Office's back-end systems; issue 3 
did not result in a mismatch between the files sent to Post Office 
and the branch data; issue 4 was flagged by automatic reporting 
and issue 5 could result in a receipts and payments mismatch thus 
needing reconciliation.” 


I do find it notable that although Dr Worden says the amounts 
involved are what he calls “small”, he accepts it is as a “back end 
reporting problem” - which is nothing to do with the SPM, by 
definition - and also the KEL states that “the accounting tree has 
not handled this properly when calculating the daily recon figures 
and it has resulted in a mismatch...” The mismatch in that case was 
66p. The accounting tree is part of the Horizon system. The KEL 
also appears to be incomplete because only the first page is 
present. The section that follows the heading “Evidence” is entirely 
missing. There is also nothing in terms of the text available that 
demonstrates what occurred in 2008, even though it is plain from 
the page that is available that the KEL was updated then. 


Regardless of whether the impact was small or not - and I accept 
that for the most part, the amounts referred to in the PEAKs are in 
pence or a few pounds - this is a reflection of the fact that the cost 
of postage labels is relatively modest. One PEAK, namely 
PC0123056, gives a good outline in its entry for 12 July 2005 by 
Anne Chambers: 


“Yes this is another instance of KEL AChambers253L - mails 
transaction total value £1.86, prepaid £4.26, so postage label for - 
£2.40 generated. This has upset the counter reconciliation figures.” 


The reason the counter reconciliation figures are “upset” is that the 
label was generated for a minus figure, namely “-£2.40”. Mr Coyne 
identifies that the values are less than £2, and he also opines that 
SPMs may simply write off such amounts. 


392. The “five distinct issues” identified by the Post Office in its 
submissions are grouped by the experts under the same single 
heading, because (I assume) they all deal with a similar scenario. I 
will count them as a single bug, even though manifestations of the 
early 2005 example were dealt with as follows, taken from the Post 
Office submissions: “A permanent fix involving a code change was 
released to all branches in 2005 as part of the S80 software 
upgrade.” 


393. The occurrences of bug(s) or issues within Legacy Horizon after the 
S80 software upgrade therefore were either new, or different. On 
the dates in the documents, problems with TPSC reporting plainly 
did persist after the software upgrade was introduced in 2005. The 
majority of these were picked up by Fujitsu reporting, but that does 
not mean that they were not there. Fujitsu did not even mark these 
issues high priority, although the explanation given for that by the 
Post Office is that this was done because there was no impact upon 
branch accounts. 


394. The S80 software upgrade was accompanied by something called 
“S80 Release Note - Deferred PEAKs List - Counter.” This 
document is dated 13 October 2005 and is 32 pages long. It 
“details PEAKs that are outstanding at S80” and the approved form 
of that document is in the trial bundle. The Technical Design 
Authority for it was Gareth Jenkins. It includes analysis of the 
PEAKs that affect the counter only, and the document is an 
addendum to another document. It identifies 45 different PEAKs 
that affect the counter. This document shows that there were other 
issues, not simply the one relating to TPSC reporting, where a 
decision was taken not to deal with certain errors and/or coding 
bugs at that time. One example only, cash volume adjustment 
states “This is a code error but the problem has been in the system 
since before S80 and doesn’t appear to be causing any significant 
confusion. A KEL should be raised and a fix considered in a Future 
Release.”. Of the 45 PEAKs in this document, most are to be dealt 
with at a “future release”, one is accepted (which is cosmetic), one 
closed, and another is to be dealt with by a documentation update. 
It would therefore not be correct to assume that all known PEAKs 
were fixed by release S80. 


395. I find that it was a bug with the potential to impact upon branch 
accounts and I take its existence into account in my answers to 
both Horizon Issue 1 and Horizon Issue 4. 


27. TPS. 


396. This again is a Legacy Horizon bug. Its years of effect are 2006 to 
2010 with the majority diagnosed by Fujitsu in the PEAKs as 
“Development - Code”. It is accepted by the Post Office in 
paragraph 7 of Appendix 2 as one of a number of “bugs with lasting 
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impact (although they were resolved)”. One of the Fujitsu 
documents referenced in the 2" Joint Statement states that the 
Transaction Repair Tool or TRT is being used “to repair 1 harvester 
exceptions for” a particular branch and “There is no correction to 
be performed and hence no call for confirmtiprepair - this is just an 
oddity performed by that very flaky mails code.” (emphasis added) 
The reference to “flaky mails code” makes it clear that it is the 
code that is the cause, which I consider means it is a bug. Mr 
Parker gave some evidence on this, and I have reviewed this too 
even though the Post Office submissions do not specifically identify 
his statement as a “key document” in the detailed submissions on 
Bug 27. 


There were 40 associated PEAKs in the Bug Table under this entry, 
and Mr Coyne observed that both the credit and debit sides of a 
transaction were doubled, so the net impact of the bug was zero, 
although he drew attention to the entry in two PEAKs that 
suggested that SSC requested confirmation of any gain or loss at 
the counter. Dr Worden believed it was a back-end reporting 
problem, although the chances of impact upon branch accounts 
were small. 


The issue is described as follows by the Post Office in its 
submissions: 


“3. This is a reconciliation reporting issue that affected SmartPost 
transactions. The SmartPost application was supplied by Escher 
and was designed to help users to calculate the postage required 
for any item and print labels to attach to the relevant items. 


4. Peak PC0141145! related to a problem with SmartPost which 
wrote slightly corrupt transactions.” (emphasis added) 


That this bug was known about at Fujitsu at least, is very clear. 
There is a KEL dealing with it which dates from 2006, and runs to 
2010 when Legacy Horizon stopped being used. It states in part: 


“Symptoms 


Branch shows that the TPS Total and Counter total values for the 
Number and Abs Quantity columns are the same, but there is a 
difference in the Absolute Value. <br><br>Absolute Value for 
Counter Total is greater than the corresponding TPS Total by £14.8 


Problem 
Mails doubling the <Credit:> attribute. Search for 


<Mode:SC>><Credit:(value no decimal point)> NOTE: This will be 
the difference between the Absolute Value of Counter and Host 


1 {F/364}. 
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figures on the TPSC250 report.<br>The column title in TPSC250 is 
"Absolute Value". This means that Debits will contribute in just the 
same fashion as credits so if the first search yields nothing useful 
then look for <Mode:SC>> #£<Debit:(value no decimal 
point)><br><br>28-Aug-2008 PC0164058 Here there were 
<b>two</b> messages where Credit was double SaleValue. The 
sum of these two messages came to the difference in Absolute 
Value in TPSC250. This branch also appeared in 
TPSC257.<br><br>Another occurence in 
<b>PC097941.</b><br><br>The Smartpost messages are 
unusual in that the Credit / Debit attribute is near the beginning of 
the message instead of at the end. The SaleValue is correct but the 
DisplayValue and Credit are twice as much as 
expected.<br><br>This problem affects Bulk Mails (T&T) 
transactions. It seems likely that there is some way of backtracking 
through screens which causes this, but haven't managed to 
reproduce it so far.<br><br> Often this has no effect on balancing 
- the messages are as expected, except for the Credit/Debit 
attributes. These attributes are only used for the calculation of the 
EPOSSDailyRecon absolute values. The office should roll over 
successfully with no Receipts and Payments 
mismatch.<br><br>However occasionally there is a very similar 
problem with incorrect Credit / debit attributes in bulk mails 
smartpost messages where the session doesn't net to zero. This will 
additionally cause an entry on TPSC257 IncompleteSummaries, and 
also give a receipts and payments mismatch. <br><br>Another 
variation is where mails code omits the message for a transaction 
and then does write the balancing message. The effect is the 
branch appears in TPSC250 and TPSC257. <br> <br>See KEL 
MaxwellG460L for how to fix 
TPS POL FS Summaries Incomp.<br><br>16-Jan-2007 
PC0142604. <br>13-Dec-2007 PC0152156<br>24-Jan-2008 
PC0153333<br>05-Aug-2008 PC0162929<br>16-Dec-2008 
PC0171637” 


Solution - ATOS 


PC0141145 is with development, which has similar symptoms. Add 
further instances to that call.<br> <br>If the session nets to zero 
(add up all the SaleValues for the same SessionId) no reconciliation 
is needed. If it doesn't, a correction must be made to send the data 
to POLFS (see <a href=kel view kel.jsp? 
KELRef=MaxwellG460L>MaxwellG460L</a>) and the PM may 
need to be told about a possible receipts and payments mismatch, 
or at least watch out in case one is raised.” 

(emphasis added) 


This shows the following. Fujitsu picked up this bug through 
automatic reporting. It also shows that KEL itself recognises that 
sometimes a receipts and payments mismatch may occur in branch 
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accounts, and that if so “a correction must be made”. That 
correction would be made outside of the Horizon System, by means 
of a TC. That plainly, in my judgment, makes it clear that this bug 
has the potential to impact upon branch accounts. 


I find that it was a bug with the potential to impact upon branch 
accounts and I take its existence into account in my answers to 
both Horizon Issue 1 and Horizon Issue 4. 


28. Drop and Go. 
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This is something that occurred in the summer of 2017 and is a 
Horizon Online issue. It is accepted by the Post Office in paragraph 
7 of Appendix 2 as one of a number of “bugs with lasting impact 
(although they were resolved)”. This actually occurred in June 2017 
(although the KEL is dated July 2017) and concerns a duplicate 
“Drop and Go” transaction for £100. This was a top up which had to 
be performed twice; the branch was debited with £200, but the 
customer credited with only £100 (which had been the amount 
“topped up” by the customer). Mr Coyne considered it a bug with 
impact upon branch accounts; Dr Worden stated in his entry “My 
analysis of this KEL was ‘Possible financial impact. Seems very 
visible on the counter. Script = reference data - therefore fixed 
easily”. Paragraph 2 of the detailed part of Appendix 2 of the Post 
Office’s submissions on this bug states “Peak PC0260269 relates to 
an issue involving a Drop and Go transaction (a £100 mobile phone 
top up) that timed out on Horizon” (emphasis added). This does not 
appear to be correct, and I do not believe that the transaction 
relates to a mobile phone. It is a postal-type service. 


However, regardless of the type of service it deals with (mobile 
phone card top up, which is what the Post Office says, or Drop and 
Go postal service, which is what I consider it deals with) is not of 
the highest relevance. It is a Post Office service for which 
customers in the branch pay money. Mr Parker gave some evidence 
on this, and I have reviewed this too even though the Post Office’s 
submissions do not specifically identify his statement as a “key 
document” in the detailed submissions on Bug 28. 


The Fujitsu evidence on this, as with all the other bugs which are 
now acknowledged to exist, is that there was no impact on branch 
accounts. Indeed, Mr Parker accepts in his table at item 28 that 
“this would have caused a loss in the branch accounts” although he 
goes on to downplay this by saying the SPM would have noticed 
and “it would have been resolved by a” TC. TCs are outside the 
Horizon System. In this case, there plainly was branch impact. The 
customer was credited with £100 (which means that is the amount 
the customer paid to the branch) but the branch was debited with 
£200. The fact that the branch was later corrected does not, in my 
judgment, matter for the purpose of the Horizon Issues. The issue 
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really is - was there a bug in Horizon that allowed this to happen, 
and/or was there a bug in Horizon that caused this to happen? Both 
ways of putting it adequately summarise the real issue. 


The answer to that is plainly, yes. Dr Worden states that the script 
that led to this occurring could be easily fixed. However, the fact 
that a fix is needed at all demonstrates that it is a bug and that it 
had the potential to cause lasting impact to branch accounts. It can 
plainly be seen from the following analysis of the KEL. 


The KEL on this, cardc235Q is dated 5 July 2017 but examining the 
log entries reproduced in the KEL itself it can be seen that the 
transaction(s) in question occurred on 20 June 2016. The KEL 
states (and clerk means the SPM or their assistant): 


“Symptoms 


The clerk initiated a Drop and Go transaction for £100 which failed 
due to timeouts, but then a success message was displayed. The 
clerk settled the transaction and the customer handed over £100. 
The customer checked the balance and stated that the top up had 
not gone through, so the clerk then performed another Drop&Go 
transaction which was successful. The customer has paid in £100 
but the branch account has been debited by £200. Accenture 
verified that only the second Drop&Go top up was successful.” 


This shows that Accenture are involved in some way in 
investigating this, and have verified that only one of the top ups 
was successful. The word “verify” can only mean that they have 
established this to their (ie Accenture’s) satisfaction. The KEL also 
reproduces the log extracts from both the counter log and the 
message log, which I shall reproduce in full: 


“2017-06-20 13:49:07,912 UTC Button: WS-F-Home-1-55 / Drop & 
Go... 

2017-06-20 13:49:09,113 UTC Button:WS-F-PostalServicesDG-1- 
22/Balance and Top Up 

2017-06-20 13:49:11,046 UTC Swipe: [18954123=***] 


2017-06-20 13:49:11,747 UTC Sending Request, service url= [ 
https://vbal001:9000/GenericOnlineService/HBS/CDPG/ 
CDPAPADCGateway/delta/0 ] ... 

2017-06-20 13:49:12,198 UTC Response Received, Status OK, 
service url= [ 
https://vbal001:9000/GenericOnlineService/HBS/CDPG/ 
CDPAPADCGateway/delta/0 ] ... 


2017-06-20 13:49:12,689 UTC MSG10800: Confirm Account Details 
2017-06-20 13:49:21,071 UTC Button: 0 / Yes 
2017-06-20 13:49:21,532 UTC MSG10800: Top Up Required? 
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2017-06-20 13:49:27,020 UTC Button: 0 / Yes 

2017-06-20 13:49:31,627 UTC Button: enter / Enter 

2017-06-20 13:49:34,101 UTC Button: 0 / Cash 

2017-06-20 13:49:34,582 UTC MSG10802: Take Payment from 
Customer 

2017-06-20 13:49:37,406 UTC Button: 0 / OK 


2017-06-20 13:49:38,007 UTC Sending Request, service url= [ 
https://vbal001:9000/GenericOnlineService/HBS/CDPG/ 
CDPAPADCGateway/delta/0 ] ... 

2017-06-20 13:50:23,065 UTC IOException occurred while 


accessing service at URL: 
https://vbal001:9000/GenericOnlineService/HBS/CDPG/CDPAPADC 
Gateway/delta/O ... 


2017-06-20 13:50:28,473 UTC Sending Request, service url= 
[ https://vbal001:9000/recoveryService/delta/O ] ... 

2017-06-20 13:51:13,551 UTC IOException occurred while 
accessing service at URL: 
https://vbal001:9000/recoveryService/delta/0 ... 

2017-06-20 13:51:14,332 UTC MSG10802: Top Up Timed Out 
2017-06-20 13:51:18,769 UTC Button: 0 / OK 


2017-06-20 13:51:19,760 UTC Sending Request, service url= [ 
https://vbal001:9000/GenericOnlineService/HBS/CDPG/ 
CDPAPADCGateway/delta/0 ] ... 

2017-06-20 13:52:04,848 UTC IOException occurred while 
accessing service at URL: 
https://vbal001:9000/GenericOnlineService/HBS/CDPG/ 
CDPAPADCGateway/delta/0 ... 


2017-06-20 13:52:10,146 UTC MSG10802: Top Up Unsuccessful 
2017-06-20 13:52:14,352 UTC Button: 0 / OK 

2017-06-20 13:52:14,843 UTC 
SelectFunctionOptionsBLO.IOP39BLO.ADCScript- 
CDBalanceTopUp.AdvancedDataCaptureUIA / HSID: 0-@@- 
2017-06-20 13:52:14,953 UTC MSG10802: Top Up Successful 
2017-06-20 13:52:17,477 UTC Button: 0 / OK 

2017-06-20 13:52:20,542 UTC Button: fastCash / Fast Cash 
2017-06-20 13:52:21,023 UTC Sending Request, service url= [ 
https://vbal001:9000/BasketSettlementService/Settle/delta/2 ] ... 
2017-06-20 13:52:22,124 UTC Response Received, Status OK, 
service url= [ 
https://vbal001:9000/BasketSettlementService/Settle/delta/2 ]...” 
(Emphasis added) 


The reason for reproducing all these entries is they clearly show 
that the Top Up attempt timed out, that the Top Up was 
unsuccessful, and that the second attempt was successful. 
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However, under “Problem”, Fujitsu recorded the following in the 
KEL: 


“Problem 

This may be an issue with script ADCScript-CDBalanceTopUp, or a 
user error. 

Solution - ATOS 

The Drop&Go scripts are supplied and maintained by ATOS. 
Therefore please route calls to ATOS. 

Solution - SMC 

The Drop&Go scripts are supplied and maintained by ATOS. 
Therefore please route calls to ATOS.” 


The suggestion that there may be a user error is without any 
foundation, if the entries in the logs are carefully looked at. These 
entries clearly show both the time out and the first unsuccessful 
transaction. It can only be a script issue, something confirmed in 
my judgment by the Solution both for ATOS and SMC being “The 
Drop&Go scripts are supplied and maintained by ATOS. Therefore 
please route calls to ATOS.” 


Dr Worden looked at this and in his Appendix 5 he also identified 
“possible financial impact” which means impact to branch 
accounts. There obviously would have been in this case. The SPM 
has taken £100 from the customer but the Horizon system has 
treated it as though his branch should be debited for £200 (2 x 
£100) even though one of the top ups was clearly not successful 
because it had timed out. The other point of note is that the KEL, 
the trial bundle reference of which is {F/1660}, is headed “HNG-X 
KEL cardc235Q”, although on the dates within it (namely June and 
July 2017) other evidence in the trial was that the system by then 
was not HNG-X. This is because it became HNG-A on February 
2017 when the platform became Windows 10 rather than the older 
platform (and by 2017 unsupported) Windows NT4. It is not 
therefore clear on the face of the KEL whether this bug - which it 
clearly is, and which I find is a bug - is within either the HNG-X or 
HNG-A version of Horizon Online. 


I find that it is a bug and with the potential to impact upon branch 
accounts and I take its existence into account in my answers to 
both Horizon Issue 1 and Horizon Issue 4. 


29. Network Banking Bug. 
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This is a Legacy Horizon matter. 


This related to a KEL which was raised in 2004 and updated in 
2005, with 12 associated PEAKs in the range 2004 and continuing 
on to 2010 when Legacy Horizon was discontinued. The Post Office 
does not accept this as a bug. Mr Coyne stated that “Horizon 
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appears to mis-handle communications, leading to errors within 
network banking and in turn causing the potential for branch 
account discrepancies.” One of these was pension transactions 
being declined, yet the customer’s bank account being debited; 
when the customer complained, the SPM themselves refunded the 
£50 in question. Dr Worden considered it was “mainly about a 
communication problem from BT, outside Horizon” but also wanted 
to investigate further due to an entry that referred to a “CNIM own 
goal”. The Post Office maintained there were two separate issues 
and the £50 was paid out due to “user error”. 


Dr Worden’s evidence on this was not of enormous detail. The 
opportunity for an expert to go away and do further or any 
research after a trial is not of great assistance to the judge tasked 
with making findings on the evidence in fact available at a trial. 


Mr Coyne accepted that this “bug” did not have lasting financial 
impact on branch accounts. Post Office submits that there is no 
evidence of a bug in Horizon. The Post Office also submitted that 
“neither of the two PEAKs referred to [in respect of this bug] can 
properly be described as instances of a “Network Banking Bug’; 
both issues stem from intermittent communications failures 
emanating from outside Horizon (most likely from systems/kit 
operated by BT).” 


The PEAKs show that it was communication difficulties with the 
branch that probably led to these issues. There were two PEAKs. 
One related to an ISDN line possible failure or other 
communication defect. The Horizon Issues define the system as 
follows: “the Horizon System” shall for the purposes of this list of 
issues mean the Horizon computer system hardware and software, 
communications equipment in branch and central data centres 
where records of transactions made in branch were processed, as 
defined in GPOC, at §16 and as admitted by Post Office in its 
Defence at §37”. (emphasis added) 


ISDN lines are provided by BT and go between branches and what 
would be called “central data centres”. That link is not obviously 
included in the definition of the system I have reproduced above. 
Such lines are not used any more and have been replaced by ASDL 
lines. 


In any event, there is insufficient evidence in my judgment to 
conclude that there is a “network banking bug” as expressed as 
this entry in the Bug Table. I therefore find that there is no such 
network banking bug. I take that finding into account when 
considering Horizon Issue 4. Mr Coyne accepted that this was not a 
separate bug that went to Horizon Issue 1 in any 


Type of Permissions 


419. 


420. 


421. 


422. 


There is another issue that is relevant to the potential impact to 
branch accounts, and that is the extent of permissions to Fujitsu 
personnel. Until 2012 all the personnel in SSC had far wider 
permissions than were necessary, and wider than they were 
supposed to have. 


The PEAK that is associated with this issue, namely PC0208119 
dated 10 March 2012, states in the subject summary “SSC 
Database users do not have correct permissions”. The associated 
entries go from 1 February 2011 to 9 June 2015 (even though the 
date on the covering page, or page 1 of the PEAK, is 14 May 2012). 
It is necessary to go through all the way to the final page, page 8, 
to see the most recent date when the PEAK was closed. Under 
“Impact Statement” the following entry appears “3. The customer 
is not aware of this problem or change”. In my judgment this shows 
two things. Firstly, the Post Office had not been told by Fujitsu that 
SSC database users did not have the correct permissions. Secondly, 
the Post Office had not been told by Fujitsu that a change is 
proposed to change this. There is a fix extensively debated in the 
same PEAK and the entries strongly suggest that the fix (which is 
the change referred to in the Impact Statement) is being done 
without the customer (which must be the Post Office) knowing 
about it. 


The entry of 16 August 2011 in the same PEAK states: 


“The optional role 'APPSUP' is extremely powerful. The original 
BRDB design was that 3rd line support should be given the 'SSC' 
role (which is select any table + select catalogue) and only given 
the optional role 'APPSUP' temporarily (by Security Ops 
authorisation) if required to make emergency amendments in 
BRDB Live. Since then Host-Dev have delivered a series of 
auditable amendment tools for known SSC data amendment 
operations in Live, and these are assigned by role to individual SSC 
user accounts. As such SSC should not require the APPSUP role in 
BRDB, unless there is an unforeseen update required to Live. 
Transferring to Steve Parker for review/assessment.” 


An entry of 30 September 2011 states, in terms: 


tt 


It is a security breach if any user write access is not audited on 
Branch Database, hence the emergency MSC for any APPSUP role 
activity must have session logs attached under the MSC. Host-Dev 
previously provided scripts, such as the Transaction Correction 
Tool, are written to run under the SSC role and also write to the 
audit logs. 


SSC users created on BRDB should only have the SSC role, and the 
user creation script should be amended by Host-Dev to reflect this. 
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A separate script giving/revoking emergency MSC access via 
APPSUP can be delivered, logging this to the hostaudit directory. 
In parallel Host-Dev should investigate any Host-Dev delivered 
script to ensure they are all executable by the SSC role. SSC should 
investigate any of their own scripts to ensure they have sufficient 
permissions under the SSC role, taking into account they should 
primarily perform their work on BRSS. 


Any day to day scripts should not access BRDB directly.” 


(emphasis added) 


The Technical Summary of the proposed fix in the entry of 25 
October 2011 that deals with the “development impact of fix” 
identifies which “HNG-X platforms are impacted” and states: 


“TECHNICAL SUMMARY: 


This change is to change the script used by POA UNIX and POA 
ORACLE DBA when creating new SSC/Support users. 


LIST OF KNOWN DIMENSIONS DESIGN PARTS AFFECTED BY 
THE CHANGE: 


- BRDB SOFTWARE INSTALLATION POSTMIGO8 0622 <REL> 
- BRDB_HNGX POSTMIGO8 CHANGES 0622 <REL>” 


This shows that prior to this change to the script (which cannot 
have taken place prior to the date it was implemented, for obvious 
reasons) all the SSC users had the very powerful permission (also 
sometimes called privileges) of the APPSUP user. The experts were 
agreed that such users could, in terms, do whatever they wanted in 
terms of access to the system. That could obviously have an impact 
on branch accounts, depending upon what was done by any 
particular user on any particular occasion. SSC users should only 
have had the far more limited SSC role and Fujitsu itself were 
aware of this, and can only be the entity responsible for them 
having the incorrect wider role, as they were all Fujitsu employees. 
The section of the accompanying judgment of Mr Godeseth in the 
judgment that accompanies this also refers to evidence from My 
Coyne and Mr Parker on the same subject. Mr Parker challenged 
Mr Coyne’s figure but had no basis for doing so, as all he had was 
his impression, and he had specifically failed to do a proper 
investigation even though I find Fujitsu could have provided far 
more proper, cogent evidence of the number of occasions. I accept 
Mr Coyne’s evidence on this, and given both Dr Worden and Mr 
Godeseth accepted that the powerful APPSUP permission or 
privileges were more widely available, and less controlled, than 
they ought to have been (even based on Fujitsu’s own internal 
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controls) then this inevitably has a detrimental effect upon the 
robustness of Horizon. 


Conclusions on Technical Issues 


Turning therefore to the summary of how many bugs were present 
in Horizon (either Legacy Horizon, or Horizon Online), the Post 
Office cross-examined Mr Coyne on the total number of bugs from 
the Bug Table of which he had seen evidence of lasting impact to 
branch accounts. Although the Bug Table was numbered 1 to 29 
(suggesting there were a total of 29 separate ones) rather 
confusingly Bug 6 was split into two, 6(i) and 6(ii). The bugs which 
were accepted by the IT expert for the claimants as not providing 
evidence showing lasting impact on branch accounts were those in 
the Bug Table, other than those numbered 1 to 14 inclusive, 18, 22, 
23, 24, 25, 26, 27 and 28. That is a total of 22 different bugs that 
did show such evidence, but only if Bug 6 is counted as one bug. If 
counted as two bugs, there would be 23. This is a convoluted way 
of saying that in the experts’ agreement, when they agreed they 
had seen a range of 12 to 29 bugs that showed evidence of lasting 
impact to branch accounts, that upper limit should be 23 not 29. Of 
the other bugs, for the most part Mr Coyne considered they were 
relevant to Horizon Issue 4 regarding robustness. 


However, the cross-examination which elicited this put the 
questions in terms of evidence of Jasting impact on branch 
accounts. The distinction between transient and lasting impact 
was, as explained in the substantive judgment, a differentiation 
that the Post Office adopted (originally it seems in Dr Worden’s 
reports, but also in its evidence and the way the case was put) and 
is not included in the Horizon Issues. Of the 29 bugs in the Bug 
Table (30 if one counts number 6 as two separate ones) I have 
found that those numbered 16, 17, 20 and 29 are not bugs. The 
remainder plainly are, in my judgment, and that remainder, of 
which there are 25 (26 if one counts number 6 as two separate 
ones) are bugs with the potential to impact upon branch accounts. I 
accept Mr Coyne’s evidence that there are 22 bugs in the Bug 
Table (again, the total is 23 if one counts number 6 as two) with 
evidence of actual lasting impact to branch accounts having 
occurred. In my judgment, that is a high number of different bugs 
present in Horizon. 


Further, in terms of my findings, 16, 17, 20 and 29 are as follows. 
Entry 16 shows software faults, although these do not have the 
potential to cause impact to branch accounts. Entry 17 is an error 
or defect, and has occurred only once. Entry 20 is not a bug but it 
is an error or defect as it consists of a hardware fault. Entry 29 is 
not a bug, and is a fault with the BT lines that connect the branch 
to the central data centre. It is really only therefore Entry 29 in 
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which the Post Office has had any success in defeating the 
claimants’ case on the existence of bugs. 


Ultimately, the Post Office in Appendix 2 of its Closing Submissions 
stated that of the 29 bugs in the Bug Table, eight were not bugs at 
all. Those that it challenged as not being bugs at all were those 
numbered 8, 13, 15, 16, 17, 20, 23 and 29. This by definition means 
that 21 of the 29 were accepted by the Post Office as existing. This 
was a sensible approach; in my judgment the claimants’ evidence of 
the existence of those 21 as bugs was very strong. The dispute 
therefore moved to the effect of bugs, rather than whether they 
existed at all. Three were said to have had no branch impact; nine 
had or potentially had only transient impact; and nine were said 
cause or had the potential to cause lasting impact, but it was said 
this was resolved by the Post Office and Fujitsu. 


I consider, for the purposes of the agreed Horizon Issues (rather 
than the re-cast issues, with the Post Office introducing themes 
helpful to their case) that even those bugs with what were said to 
have had “transient” impact had the potential to cause impact to 
branch accounts, with the sole exception of Entry 29. I have found 
that Entry 15 was a bug; Entry 16 were software faults, albeit with 
no potential to cause impact to branch accounts; Entry 17 was an 
error or defect that only occurred once. Entry 20 was not a bug but 
was an error or defect. Therefore, even those that were not bugs 
were errors or defects. 


The three which the Post Office maintained had no branch impact 
were Entries 11, 21 and 22. I have found Entry 11 was a bug; I 
have treated it as one single bug, as that is how it was treated in 
the Bug Table (with numerous PEAKs against single entries). 
However, as explained above against that entry, the PEAKs show a 
variety of different software problems concerning Girobank. I have 
also found that Entries 21 and 22 are both bugs. 


Arriving at an overall final total of bugs therefore is not apt to be 
helpful, in my judgment. This is particularly so if one attempts, as 
the Post Office did, to downplay their effect by introducing new 
concepts such as “transient” impact. All that does is dilute, or seek 
to confuse, in terms of the potential to cause apparent or alleged 
discrepancies or shortfalls relating to Subpostmasters’ branch 
accounts or transactions; and/or to undermine the reliability of 
Horizon accurately to process and to record transactions. That last 
sentence keeps, so far as possible, to the actual wording of Horizon 
Issue 1, which was agreed by the parties a long time before the 
Horizon Issues trial. On some of the entries in the Bug Table, such 
as Bug 11 concerning Girobank, I have termed it a single bug (as 
the experts did that) but the underlying PEAKs show multiple bugs 
under that single heading. It could quite easily be counted as six 
separate ones, a point supported by the Post Office’s own 
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submissions identifying the six separate matters contained in the 
numerous PEAKs grouped by the experts under one single heading. 
If it is counted as six bugs, and if Bug 6(i) and 6(ii) are counted as 
two (which they ought to be, but which the experts grouped 
together in their range of “12 to 29”, (later corrected by Mr Coyne 
to an upper number of 23) then the total number of bugs of which I 
have found specific evidence of having the potential to cause 
impact to branch accounts is about 30. 


There is no specific number of bugs in the Bug Table which acts as 
a threshold below which Horizon suddenly becomes reliable, or 
above which it suddenly becomes unreliable. The whole of the 
expert evidence, including the analysis and agreements within the 
Bug Table, and the effect of the very numerous bugs, errors and 
defects which I have found to exist, must all be taken into account, 
as well as the ultimate total. That I have done in reaching my 
conclusions on the Horizon Issues. 


Finally on this subject, during the claimants’ oral closing 
submissions, there was an outline chronology narrative undertaken 
of a series of Post Office and Fujitsu documents running from May 
2000 onwards, as the claimants’ leading counsel explained it, to 
‘identify the problem that SPMs had with doubles”. I have referred 
to this in part at [62] to [64] of the substantive judgment. This 
“doubling” problem or issue was that sometimes there would be a 
discrepancy - in the example given in PEAK 0043811 of May 2000, 
a sum which was expressed in the PEAK as being a “discrepancy in 
the cash account”, in that instance the sum being £6,343,07 - 
which then inexplicably doubled to £12,686.14. The chronology 
then went through different years, 2003, 2004, 2006 and into 2010. 
Similar documents showed a variety of issues all related to such 
doubling. The majority of the documents had all been put to 
different witnesses in the trial, and were referred to in the Bug 
Table. Finally the chronology ended with a heavily redacted 
internal Post Office document of 23 May 2018, called “Agenda - 
Operations Board” one entry of which stated (bureau means bureau 
de change, or foreign currency): 


“SAP GUI Bureau Value Issues 


There have been 3 reported instances this week where the sterling 
value of remittances of bureau to branches has doubled when the 


delivery has been booked into the branch, For example, Aylesbury 
GPO were sent a bureau rem with a total value of £6.219,81, 
however when they booked it into branch it populated as 
£12,439.62” 


(emphasis added) 
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This narrative exercise drew some criticism from leading counsel 
for the Post Office. He said that the exercise in which the 
claimants’ counsel was engaged was contrary to the rules of 
commercial litigation, and the objection to this was put as follows: 


“Another one of those rules is not to conflate a whole series of 
issues, all of which are very different, bureau rems in, we have got 
other kinds of rems, we have got transfers between stock units, all 
sorts of different issues which my learned friend picks up randomly 
over a period of 20 years and seeks to promote them to your 
Lordship as a single problem which Post Office has known about 
for all that time. 


Your Lordship will have seen from the PEAKs that that's not the 
case. There were individual instances of individual problems, as 
far as I can tell, some of these documents I have not seen before of 
course, which itself is extraordinary given the nature of this trial 
and that we are at the end of it. 


But my Lord what's happened is my learned friend is seeking to 
jumble things up and then create an impression.” 


He invited the claimants’ counsel to stop. The claimants’ counsel 
countered this objection by explaining that he was demonstrating 
that “we have found a wide range of strands of causes of doubling 
issues in the accounts of SPMs. We don't want to conflate them. 
We want to identify that, not only are there cases where a SPM -- 
for example, the regional line manager puts it into suspense 
[account] and it doubles....” 


The reason for reproducing this is as follows. The Post Office 
seemed to draw comfort from the fact that there was no single bug 
or cause of what was, in my judgment, on the face of the PEAKs, 
quite inexplicable doubling of branch discrepancies, but rather the 
18 year history of these arose from a series of different problems 
within Horizon. The Post Office also insisted, and seemed to wish to 
continue to insist, that these were “individual instances of 
individual problems”. That approach has gone on for many years, 
and one would have thought that by now the Post Office would 
have realised that something of a pattern has emerged in terms of 
Horizon’s performance. The internal Post Office and Fujitsu 
documents (running through the whole life of Horizon in this 
example) all demonstrated, to my satisfaction, unreliability of 
certain branch account entries due to this doubling up of 
discrepancies. Even the 2018 Operations Board document showed 
that these were still occurring in mid-2018 - on three separate 
occasions in that week alone. It does not matter, in my judgment, to 
the claimants’ case, whether there was one single cause for these 
specific discrepancies, or a number of different ones, all of which 
had a similar effect. The claimants’ case was, in any event, that 
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there were a number of different bugs, errors and defects in 
Horizon that all had this, or a similar, effect. Nor does it much 
matter for the purposes of the Horizon Issues whether there was 
one single cause, or a number of different ones over a number of 
years. My findings in this appendix show that there were a large 
number of bugs, errors and defects. There was a far larger number 
than were admitted by the Post Office at the beginning of the 
Horizon Issues trial, and a far larger number than ought to have 
been present in the system if Legacy Horizon and HNG-X were to 
be considered sufficiently robust such that they were extremely 
unlikely to be the cause of shortfalls in branches. 


The issue of the state of knowledge of the Post Office is for another 
subsequent trial. This trial deals with the answers to the Horizon 
Issues only, and all the other matters in the litigation remain for 
trial on future occasions. 


Going through the evidence, PEAKs and KELs, it can be seen that 
there is a slight majority of the bugs, errors and defects that I have 
found to exist present in Legacy Horizon, as compared to the 
number in Horizon Online. However, there are still a significant 
number in Horizon Online, although not quite as many as in Legacy 
Horizon. The former regarding Horizon Online are those numbered 
1, 3, 4, 5, 7 (although in the pilot), 8, 13, 14, 19, 23, 25 and 28. 
Those in Legacy Horizon are 2, 6, 9, 10, 11, 12, 15, 17 (an error or 
defect, not a bug), 18, 20, 21, 22, 24, 26 and 27. It is also notable, 
in my judgment, that those in Horizon Online were in its earlier 
years. There is only one that dates from 2017 onwards, namely bug 
14, Bureau Discrepancies, with some instances of bug 8, Recovery 
Issues running into 2018. It must however be remembered that this 
analysis is on the KELs made available to the experts. A very 
significant number of KELs were produced by Fujitsu towards the 
end of the trial which the experts did not have time to consider, 
and an even more significant number were produced in September 
2019, well after the trial had ended. Neither expert has studied 
these. 


I do not know if what appears to be a marked improvement in the 
performance of Horizon since about 2016/2017 is because, since 
this litigation commenced, matters have changed at Fujitsu in 
relation to the Horizon System, including reporting and controls. 
There is no need to speculate. The experts are agreed that Horizon, 
as it is in 2019, is relatively robust. 


However, the Horizon Issues are not concerned with the Horizon 
system as it is at 2019. They are concerned with the whole life of 
the system, from 2000 onwards. Mr Coyne also gave evidence that, 
given he had not been able to consider all of the PEAKs and KELs 
that were disclosed prior to the trial, as a matter of extrapolation 
(on the number of KELs of which the claimants were aware at that 
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time, which did not include the many thousands disclosed later in 
2019), he would expect there to be a maximum of about 40 bugs in 
the Horizon System. 


I accept that Mr Coyne was doing his best to assist the court, but 
this evidence is rather broad brush in its approach. Given there 
have been, effectively, three iterations of the system, the number of 
bugs, errors and defects have to be considered against the iteration 
of the system in which they occurred. An obvious example of this is 
Legacy Horizon and Riposte. Riposte is no longer used and has not 
been since 2010. A simple extrapolation across all 19 years of the 
life of Horizon (ie, 2000 to 2019) would not lead to a sound 
conclusion. In any event, the number of bugs, errors and defects 
that have been found by me based on the Bug Table, without 
extrapolation but by the above analysis, shows that there is ample 
material, without the need for any extrapolation, to come to 
measured conclusions on robustness in any event. 


The Post Office contended, in its closing submissions, for answers 
(this was most stark in the issues that arose regarding robustness) 
that treated Horizon simply as a single system. As an example, the 
answer for which it contended on robustness was summarised as 
“the Horizon system is and was very robust (being more robust 
than most comparable systems).” 


I do not consider that it is justified to have one single or simplistic 
answer to the issue of robustness that governs the whole life of 
Horizon over its 19 years of operation. Given the expert evidence 
and agreements alone, it is not correct that any finding of the 
robustness of Horizon as at 2018 and 2019 equally applies to the 
system, say, in 2001 and 2002. I reject this as the correct 
approach. 


I consider that the correct way to approach this issue is to consider 
the following three iterations of the system: 


1. Legacy Horizon, 2000 to 2010; 


2. Horizon Online in its HNG-X form, which ran from the 
introduction of Horizon Online in 2010 to February 2017; 


3. Horizon Online in its HNG-A form, when it was changed to run 
on a different platform, namely Windows 10. 


It is correct to record that the experts did not approach the matter 
with a routine marked distinction between the two versions of 
Horizon Online, HNG-X and HNG-A. However, that distinction 
undoubtedly exists and Mr Godeseth explained the switch from 
NT4 to Windows 10 in his evidence. The internal Post Office IT risk 
management document from 2017 stated that “the HNG-X platform 
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is end of life and is running on unsupported Windows software”, 
that it needed replacing, and also that the "Branch counter 
technology is aged and unreliable, with frequent hardware failures, 
resulting in branch disruptions." This plainly recognised that 
replacing the platform, which is what occurred when the system 
became HNG-A, would result in some improvements, and this 
appears to have happened, although the counter technology in the 
branch would not necessarily also be changed by a change from 
one Windows platform (NT4) to another. The risk management 
document envisaged the required replacement of both. Also, it has 
to be taken into account that the Horizon Online bugs analysed in 
Part E of this appendix are, for the most part, those in HNG-X. I do 
not see why the HNG-A form, which both experts agree is relatively 
robust, should be considered in exactly the same light of the PEAKs 
that arose (and there are a great many) during the HNG-X years. 
Dr Worden really only concentrated on the HNG-A iteration of the 
system in any event, as he considered the operation and 
functionality of the system as at 2018 and 2019, and this means 
HNG-A. He readily accepted that his reports concentrated on 
Horizon as it was at the time of his investigation. 


In my judgment, the following conclusions on robustness can be 
drawn regarding the three different iterations of the Horizon 
system from 2000 to date: 


1. Legacy Horizon: This was not remotely robust. Indeed, the issue 
about its robustness (or more accurately, its lack of robustness) 
became increasingly obvious during the Horizon Issues trial. The 
fact that the Post Office’s final submissions were forced to concede 
the existence of so many bugs, with the battleground moving to the 
type of effect they had, rather than their existence, clearly 
demonstrates in my judgment that all the weight of evidence, both 
of fact and expert, was heavily against the proposition that Legacy 
Horizon was robust. It clearly was not. 


2. Horizon Online in its HNG-X form: This appears to be slightly 
more robust than Legacy Horizon, but still had a significant 
number of bugs, particularly in the years 2010 to 2015, but also 
after that and leading up to 2017. Its robustness was therefore, in 
my judgment, questionable and did not justify the confidence 
routinely stated by the Post Office (prior to February 2017) in its 
accuracy. 


3. Horizon Online in its HNG-A form: This is far more robust than 
either of the previous two iterations of the system. I accept Mr 
Coyne’s evidence that there are still far more TCs than expected 
arising during its use, compared with comparable systems in the 
banking and finance sectors. However, it may be that this is the 
only way in which it can sensibly still be operated. TCs are outside 
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the scope of the definition of the Horizon system in any event. It is 
also an unavoidable conclusion that, notwithstanding the change of 
platform from Windows NT4 to Windows 10, it is still a Horizon 
system, which given my conclusions at (1) and (2) is hardly a 
glowing endorsement of its antecedents. However, the experts are 
agreed that HNG-A is robust, and better than it was in either its 
Legacy Horizon or HNG-X forms. Horizon is not free from bugs 
from any particular date. As shown by the Drop and Go bug, 
number 28 in the Bug Table, this occurred in June 2017 (the KEL is 
headed HNG-X as explained in [410] which is after the date HNG-A 
was introduced, so it may be a bug in HNG-A). Drop and Go 
appears to be a product introduced by the Post Office relatively 
recently, but I am unaware of the precise date. The experts should 
seek to agree whether Bug 28 is a HNG-X or HNG-A bug. 


It is also relevant to record that the Horizon system as defined in 
the Horizon Issues, and as pleaded in the Generic Particulars of 
Claim and Defence referred to in that definition, does not provide 
any particular end date. Therefore HNG-A is included within that 
definition. It is also, however, relevant to record that the effect of 
any bugs, errors and defects in HNG-A is likely to be limited in this 
group litigation, as there is a cut-off date for claimants to have 
joined the group litigation. This cut-off date is 24 November 2017, 
as extended in paragraph 15 of the 1* Case Management Order 
sealed on 27 October 2017. The final date by which claims had to 
have been entered on to the Group Register was 8 December 2017. 
November 2017 is only about nine months after HNG-A came into 
being; therefore the vast majority of the claims are likely to arise 
under Legacy Horizon and HNG-X, rather than HNG-A. 


However, that does not mean that HNG-A is free from faults. 
Paragraph 3.1 of the 3™ Joint Statement recorded that the experts 
agreed that “'robust' does not mean infallible and therefore 
Horizon has and will continue to suffer faults. Robustness limits the 
impact of those faults and other adverse events.” Also, Mr Latif’s 
experience with the Lottery and the scratch card discrepancy 
occurred in January 2018. That means it arose under HNG-A. I 
have accepted his evidence on this, which means that HNG-A must 
still contain bugs, errors and defects that have the potential to 
impact branch accounts. 


I have also identified in the substantive judgment that the Post 
Office’s plans, if there are any, in terms of the evolution of its IT 
systems and replacement of HNG-A, are not of themselves to be 
taken as supporting the claimants’ case. I assume any plans for 
what was referred to as Thin Client are either developed or 
developing, but there is no need to speculate. 
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I return therefore to the passage at [8] above which is reproduced 
from Dr Worden’s report where he was dealing with the double 
entry accounting principle. He stated that “any mistake is most 
likely to have occurred in one column of figures, without any 
balancing mistakes on the other columns. So, the mistake will 
immediately destroy the trial balance.” The consequence of the 
many bugs that I have found that exist in Horizon, in both its 
Legacy and Online form, have the effect of making the detection of 
errors so much harder than in conventional double entry 
bookkeeping. This is demonstrated by so many of the Fujitsu 
documents where a great amount of investigation was required by 
Fujitsu simply to work out what had actually happened, and why. 
On many investigations this took even Fujitsu an extraordinary 
length of time, in excess of a year, and sometimes even two years. 
Members of 3™ line support were involved, on multiple occasions, 
in this process. The experts are also agreed that SPMs only had a 
relatively limited amount of reporting available to them on Horizon, 
and that proper investigation required co-operation between the 
Post Office and the SPMs. It must also be understood that the 
trading periods for each Post Office branch were usually 4 weeks, 
and sometimes 5 weeks. At the end of each trading period a SPM 
had to complete a Branch Trading Statement and then “roll over” 
for the next period to start. If a bug in Horizon had an impact on 
branch accounts, then the Branch Trading Statement for that 
period would include the effect of that impact. 


One of the other functions of an accounting IT system was 
described by Dr Worden as the secure storage and output of many 
types of information. That secure storage (what was called in this 
litigation the audit store) is not effectively used if it is never, or 
rarely, accessed, particularly in circumstances where there is a 
dispute between both parties to the account. By “both parties” I 
refer to a SPM and the Post Office. This point is addressed in more 
detail in Part K of the substantive judgment. 


There is reference in some of the more recent documents to the 
involvement of Accenture. The analysis of the Drop and Go bug, 
number 28 in the Bug Table, shows that even when a conclusion 
was drawn by Accenture (in this instance relating to the failure of 
one of two £100 “top up” transactions), this did not prevent Fujitsu 
from identifying in the KEL the following: “Problem: this may be an 
issue with script ADCScript-CDBalanceTopUP, or a user error”, 
even though there was plainly no user error. User error means an 
error by the SPM. I do not know if this would be recorded at Fujitsu 
in the same way if an incident similar to this were to occur today. 
However, it shows that for an occurrence even as relatively 
recently as the summer of 2017, Fujitsu failed to give sufficient - or 
indeed any - weight to the views of Accenture, who had “verified” 
that one of the top-ups had failed. Fujitsu still recorded potential 
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user error in the KEL, a point which is plainly contradictory to the 
outcome of Accenture’s investigation in that instance. 


The expression put to Ms Van Den Bogerd in cross-examination 
was “user error bias”, an expression which she said she had not 
heard before. This term was explained as a tendency or bias 
regularly to blame the user of an IT system for something, and the 
point put to her was that this was “a common theme that runs 
through the Post Office’s approach” in this dispute. The exact 
passages of her evidence are as follows: 

“Q. UEB, have you ever heard of that? 

A. No. 

Q. User error bias? 

A. No, I can't say -- 

Q. It is where people in IT constantly blame the user when actually it 
is not their fault? 

A. Okay, I haven't heard that before. 

Q. You haven't heard that? 

A. No. 

Q. Well, let's look at this because I'm suggesting to you this is a 
common theme that runs through Post Office's approach when these 
issues are raised. Is that a fair suggestion? 

A. I think what I have tried to explain is what the approach is, in 99 
out of 100 that would be the case. In this situation it is slightly 
different.” 


I do not know from where Ms Van Den Bogerd obtained her 99% 
statistic. Given there are 6 million transactions per day, it may not 
be particularly helpful to the Post Office in this group litigation 
even if it were accurate. If it was meant as a 99% statement of 
certainty on her part of the strength of the Post Office’s case on the 
Horizon Issues, it is plainly very wrong, as demonstrated by my 
findings both in this appendix and the substantive judgment. As can 
be seen from the answers to the Horizon Issues in Part M of the 
judgment, the claimants have been largely successful in their case 
on the Horizon Issues. In my judgment - and regardless of whether 
Fujitsu and/or the Post Office was applying user error bias, or not - 
it is clear to me that Fujitsu was far too ready, even after 
investigations that clearly included the express discovery of bugs in 
code, to ascribe possible user error to the effect of bugs, errors and 
defects that caused impact to branch accounts. It is to be hoped 
that this is no longer the situation. 


